Jul 28 17:14:58.433006 ip-10-0-137-64 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 17:14:58.433193 ip-10-0-137-64 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 17:14:58.433297 ip-10-0-137-64 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 17:14:58.433719 ip-10-0-137-64 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 17:15:08.682088 ip-10-0-137-64 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 17:15:08.682105 ip-10-0-137-64 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 2cd12c5d0e3a4e16b51363dd7747e05b --
Jul 28 17:17:02.950942 ip-10-0-137-64 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 17:17:03.352301 ip-10-0-137-64 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:03.352301 ip-10-0-137-64 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 17:17:03.352301 ip-10-0-137-64 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:03.352301 ip-10-0-137-64 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 17:17:03.352301 ip-10-0-137-64 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 17:17:03.352930 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.352377    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 17:17:03.358139 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358124    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:03.358139 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358138    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358144    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358149    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358152    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358155    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358158    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358161    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358164    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358166    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358170    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358173    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358175    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358178    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358181    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358183    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358186    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358189    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358192    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358195    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:03.358203 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358197    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358200    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358204    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358207    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358215    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358218    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358220    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358223    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358226    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358228    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358231    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358235    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358238    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358241    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358244    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358247    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358250    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358252    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358255    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:03.358645 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358258    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358261    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358264    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358266    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358269    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358272    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358274    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358277    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358279    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358282    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358284    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358287    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358289    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358292    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358294    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358298    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358301    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358303    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358306    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358309    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:03.359150 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358312    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358316    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358319    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358322    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358325    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358328    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358330    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358333    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358335    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358338    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358340    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358343    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358346    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358349    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358351    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358354    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358356    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358359    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358361    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358364    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:03.359628 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358366    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358369    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358371    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358374    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358376    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358379    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358381    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358384    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358387    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358390    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358392    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358395    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358398    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358400    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358403    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358405    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358408    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358411    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358413    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.358416    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:03.360121 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359166    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359173    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359176    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359179    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359182    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359185    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359188    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359191    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359193    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359196    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359199    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359201    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359204    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359207    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359211    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359215    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359218    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359221    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359223    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:03.360599 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359227    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359230    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359232    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359235    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359238    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359240    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359243    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359246    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359248    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359251    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359253    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359256    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359259    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359261    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359264    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359266    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359269    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359272    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359274    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359277    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:03.361088 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359280    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359282    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359285    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359287    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359290    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359292    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359296    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359299    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359303    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359305    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359308    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359310    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359314    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359317    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359320    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359323    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359326    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359328    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359331    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359334    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:03.361567 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359336    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359339    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359341    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359344    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359347    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359349    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359353    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359355    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359358    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359360    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359363    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359365    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359368    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359371    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359374    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359376    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359379    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359381    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359384    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359386    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:03.362092 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359388    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359391    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359394    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359396    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359399    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359402    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359405    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359408    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359410    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359413    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359415    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359418    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359422    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359425    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359429    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359432    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359434    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359437    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359440    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:03.362579 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359442    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359811    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359816    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359819    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359822    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359825    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359828    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359830    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359833    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359836    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359839    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359841    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359847    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359849    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359852    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359854    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359857    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359860    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359862    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359865    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:03.363064 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359867    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359870    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359873    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359875    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359878    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359880    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359883    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359887    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359891    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359894    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359896    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359899    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359902    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359904    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359907    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359910    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359912    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359914    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359917    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:03.363552 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359920    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359923    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359927    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359930    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359934    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359938    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359941    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359943    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359946    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359948    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359951    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359954    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359956    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359959    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359961    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359964    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359966    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359969    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359971    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359975    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:03.364030 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359977    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359980    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359983    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359985    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359987    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359990    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359992    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359995    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.359998    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360001    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360003    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360006    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360009    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360014    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360017    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360021    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360023    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360027    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360030    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360032    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:03.364499 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360035    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360038    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360041    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360044    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360047    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360049    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360051    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360054    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360056    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360059    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360061    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360065    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360069    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360071    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360074    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360076    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360079    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360082    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360085    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360087    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:03.364996 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360090    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360152    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360159    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360165    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360170    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360174    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360177    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360182    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360186    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360190    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360194    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360197    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360201    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360204    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360206    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360209    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360212    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360215    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360218    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360222    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360225    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360228    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360231    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360235    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 17:17:03.365478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360238    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360241    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360244    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360247    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360250    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360253    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360256    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360259    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360263    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360267    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360270    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360273    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360275    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360279    2563 flags.go:64] FLAG: --enable-server="true"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360283    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360287    2563 flags.go:64] FLAG: --event-burst="100"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360290    2563 flags.go:64] FLAG: --event-qps="50"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360294    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360297    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360302    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360305    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360309    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360311    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360314    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360317    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 17:17:03.366065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360320    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360323    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360326    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360328    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360331    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360334    2563 flags.go:64] FLAG: --feature-gates=""
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360338    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360341    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360344    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360347    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360350    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360353    2563 flags.go:64] FLAG: --help="false"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360356    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360359    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360362    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360365    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360368    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360371    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360374    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360377    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360380    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360383    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360386    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360389    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 17:17:03.366723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360393    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360396    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360400    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360403    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360406    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360409    2563 flags.go:64] FLAG: --lock-file=""
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360412    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360415    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360418    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360423    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360426    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360429    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360432    2563 flags.go:64] FLAG: --logging-format="text"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360435    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360438    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360441    2563 flags.go:64] FLAG: --manifest-url=""
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360443    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360448    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360451    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360455    2563 flags.go:64] FLAG: --max-pods="110"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360457    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360460    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360463    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360466    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360469    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 17:17:03.367411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360472    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360475    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360481    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360484    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360487    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360490    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360493    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360498    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360503    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360507    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360510    2563 flags.go:64] FLAG: --port="10250"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360513    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360516    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0d9f0e127f02477e2"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360519    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360522    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360525    2563 flags.go:64] FLAG: --register-node="true"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360528    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360531    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360534    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360537    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360540    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360543    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360546    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360549    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360552    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 17:17:03.368022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360555    2563 flags.go:64] FLAG: --runonce="false"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360558    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360561    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360564    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360567    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360569    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360572    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360575    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360578    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360581    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360584    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360587    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360590    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360592    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360596    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360600    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360607    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360610    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360613    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360617    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360619    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360622    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360625    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360628    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360631    2563 flags.go:64] FLAG: --v="2"
Jul 28 17:17:03.368624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360635    2563 flags.go:64] FLAG: --version="false"
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360639    2563 flags.go:64] FLAG: --vmodule=""
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360642    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.360646    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360778    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360782    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360785    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360788    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360790    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360793    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360796    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360798    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360801    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360803    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360806    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360808    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360811    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360814    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360816    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360819    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:03.369247 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360821    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360824    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360834    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360839    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360841    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360844    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360847    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360849    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360852    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360854    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360857    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360859    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360862    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360864    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360867    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360869    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360872    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360875    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360877    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360880    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:03.369743 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360882    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360885    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360887    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360889    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360892    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360894    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360897    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360899    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360902    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360904    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360907    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360909    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360912    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360914    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360917    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360928    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360931    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360935    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360939    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:03.370226 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360942    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360945    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360948    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360951    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360954    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360957    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360960    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360962    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360965    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360968    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360971    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360973    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360976    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360978    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360981    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360983    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360986    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360988    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360991    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360993    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:03.370723 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360996    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.360998    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361001    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361003    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361006    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361008    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361011    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361014    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361018    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361026    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361028    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361031    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361033    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361036    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361039    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361041    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361045    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361048    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361052    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:03.371230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361055    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361058    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361061    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361063    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.361066    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.361854    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.368010    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.368023    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.368043    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.368049    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368107    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368112    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368116    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368119    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368122    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:03.371702 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368125    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368128    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368130    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368133    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368136    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368138    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368141    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368144    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368148    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368150    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368153    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368155    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368158    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368160    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368163    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368165    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368168    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368171    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368174    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368177    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:03.372070 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368179    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368182    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368184    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368187    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368191    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368194    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368196    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368199    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368203    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368207    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368210    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368213    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368216    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368219    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368222    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368225    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368228    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368231    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368234    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368237    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:03.372546 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368240    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368244    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368246    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368249    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368252    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368255    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368257    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368260    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368262    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368265    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368267    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368270    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368272    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368275    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368278    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368280    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368283    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368286    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368289    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368292    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:03.373039 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368296    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368300    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368302    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368304    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368307    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368310    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368312    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368314    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368317    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368320    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368323    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368325    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368328    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368331    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368333    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368336    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368338    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368341    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368343    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:03.373525 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368346    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368349    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368351    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368354    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368356    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368359    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368361    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368363    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368366    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368369    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368372    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368375    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368378    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368380    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:03.374110 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368384    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.368389    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368495    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368499    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368502    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368505    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368507    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368510    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368512    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368515    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368518    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368520    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368523    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368526    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368528    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 17:17:03.374481 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368531    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368534    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368536    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368539    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368542    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368544    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368547    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368550    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368552    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368555    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368557    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368560    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368563    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368565    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368568    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368570    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368573    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368575    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368578    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368580    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 17:17:03.374877 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368583    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368586    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368588    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368590    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368593    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368595    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368598    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368600    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368603    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368605    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368608    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368612    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368615    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368618    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368620    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368623    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368625    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368628    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368631    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 17:17:03.375444 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368634    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368636    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368639    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368642    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368646    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368649    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368651    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368654    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368657    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368659    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368662    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368664    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368667    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368682    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368685    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368687    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368690    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368692    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368695    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368697    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 17:17:03.375938 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368700    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368702    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368705    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368707    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368710    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368713    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368715    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368718    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368721    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368723    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368726    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368728    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368731    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368733    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368736    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368738    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368741    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368744    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368747    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 17:17:03.376545 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368749    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368752    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368754    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368757    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368759    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368762    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368765    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:03.368769    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.368773    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.368882    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.370786    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.371907    2563 server.go:1019] "Starting client certificate rotation"
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.372003    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:17:03.377027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.372037    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 17:17:03.377348 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.374378    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:17:03.380857 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.380822    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 17:17:03.392575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.392555    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 28 17:17:03.394509 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.394488    2563 log.go:25] "Validated CRI v1 image API"
Jul 28 17:17:03.395305 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.395292    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 17:17:03.401894 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.401874    2563 fs.go:135] Filesystem UUIDs: map[5bed48cb-f03b-4c62-8f99-47ed367f54d2:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 f8780e7f-567d-4e57-985f-e5f083b1f589:/dev/nvme0n1p4]
Jul 28 17:17:03.401954 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.401894    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 17:17:03.404547 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.404528    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:17:03.406772 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.406635    2563 manager.go:216] Machine: {Timestamp:2026-07-28 17:17:03.405409027 +0000 UTC m=+0.338808436 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101775 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2c9d141bade94519d66f9fead33d8a SystemUUID:ec2c9d14-1bad-e945-19d6-6f9fead33d8a BootID:2cd12c5d-0e3a-4e16-b513-63dd7747e05b Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c0:67:ae:96:69 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c0:67:ae:96:69 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:92:12:77:9f:32:54 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 17:17:03.406772 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.406764    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 17:17:03.406891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.406860    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 17:17:03.409842 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.409819    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 17:17:03.409971 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.409845    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-137-64.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 17:17:03.410582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.410572    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 17:17:03.410611 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.410584    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 17:17:03.410611 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.410591    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:17:03.411333 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.411323    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 17:17:03.412010 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.412001    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 17:17:03.413489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413479    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:17:03.413600 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413591    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 17:17:03.413638 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413619    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 17:17:03.413638 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413628    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 17:17:03.413713 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413641    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 17:17:03.413713 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413650    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 28 17:17:03.413713 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413659    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 17:17:03.413989 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413978    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:17:03.414027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.413997    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 17:17:03.416305 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.416278    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 17:17:03.418158 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.418144    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 17:17:03.418215 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.418173    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 17:17:03.419485 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419471    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 17:17:03.419532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419493    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 17:17:03.419532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419501    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 17:17:03.419532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419508    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 17:17:03.419532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419518    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 17:17:03.419532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419523    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 17:17:03.419532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419531    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 17:17:03.419701 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419536    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 17:17:03.419701 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419542    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 17:17:03.419701 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419548    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 17:17:03.419701 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419556    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 17:17:03.419701 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.419561    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 17:17:03.420305 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.420294    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 17:17:03.420305 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.420304    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 17:17:03.423261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.423247    2563 server.go:1268] "Started kubelet"
Jul 28 17:17:03.423348 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.423328    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 17:17:03.425138 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.425110    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 17:17:03.425303 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.425282    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 17:17:03.425960 ip-10-0-137-64 systemd[1]: Started Kubernetes Kubelet.
Jul 28 17:17:03.427251 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.427234    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 17:17:03.428195 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.428177    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 17:17:03.428286 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.428250    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-137-64.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 17:17:03.428408 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.428360    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 17:17:03.428642 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.428621    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-137-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 17:17:03.432632 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.431547    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-137-64.ec2.internal.18c6839f8333af6a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-137-64.ec2.internal,UID:ip-10-0-137-64.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-137-64.ec2.internal,},FirstTimestamp:2026-07-28 17:17:03.42322161 +0000 UTC m=+0.356621019,LastTimestamp:2026-07-28 17:17:03.42322161 +0000 UTC m=+0.356621019,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-137-64.ec2.internal,}"
Jul 28 17:17:03.432872 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.432854    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 17:17:03.435934 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.435920    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pzbzx"
Jul 28 17:17:03.437003 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.436988    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 17:17:03.437003 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.436997    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 17:17:03.437114 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.437052    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:03.437517 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.437470    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 17:17:03.437663 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.437641    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 17:17:03.437749 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.437665    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 17:17:03.437822 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.437808    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 17:17:03.437822 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.437822    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 17:17:03.438310 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.438294    2563 factory.go:55] Registering systemd factory
Jul 28 17:17:03.438367 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.438334    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 28 17:17:03.439789 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.439777    2563 factory.go:153] Registering CRI-O factory
Jul 28 17:17:03.439914 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.439904    2563 factory.go:223] Registration of the crio container factory successfully
Jul 28 17:17:03.439983 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.439971    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 17:17:03.440043 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.439995    2563 factory.go:103] Registering Raw factory
Jul 28 17:17:03.440043 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.440015    2563 manager.go:1196] Started watching for new ooms in manager
Jul 28 17:17:03.440466 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.440453    2563 manager.go:349] Starting recovery of all containers
Jul 28 17:17:03.444169 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.444146    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-pzbzx"
Jul 28 17:17:03.450977 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.450961    2563 manager.go:354] Recovery completed
Jul 28 17:17:03.451512 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.451494    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 17:17:03.451656 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.451624    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-137-64.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 17:17:03.454725 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.454664    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:03.456400 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.456386    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:03.456452 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.456414    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:03.456452 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.456427    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:03.456894 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.456878    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 17:17:03.456894 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.456892    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 17:17:03.457006 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.456901    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:03.457006 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.456907    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 17:17:03.458124 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.458110    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:03.458177 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.458135    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:03.458177 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.458144    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:03.458446 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.458435    2563 policy_none.go:49] "None policy: Start"
Jul 28 17:17:03.458475 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.458451    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 17:17:03.458475 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.458461    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 17:17:03.459270 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.459260    2563 policy_none.go:47] "Start"
Jul 28 17:17:03.502290 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.502183    2563 manager.go:337] "Starting Device Plugin manager"
Jul 28 17:17:03.502370 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.502302    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 17:17:03.502370 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.502312    2563 server.go:87] "Starting device plugin registration server"
Jul 28 17:17:03.502533 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.502521    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 17:17:03.502570 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.502533    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 17:17:03.502660 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.502643    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 17:17:03.502823 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.502752    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 17:17:03.502823 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.502764    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 17:17:03.503286 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.503268    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 17:17:03.503341 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.503321    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:03.589559 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.589515    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 17:17:03.590685 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.590654    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 17:17:03.590744 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.590720    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 17:17:03.590781 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.590747    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 17:17:03.590816 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.590786    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 17:17:03.593029 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.593008    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:03.603022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.602990    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:03.603900 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.603883    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:03.603959 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.603916    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:03.603959 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.603931    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:03.603959 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.603950    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.612840 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.612819    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.612885 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.612846    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-137-64.ec2.internal\": node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:03.630450 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.630433    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:03.691492 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.691477    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal"]
Jul 28 17:17:03.691552 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.691529    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:03.694803 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.694789    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:03.694862 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.694815    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:03.694862 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.694828    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:03.695998 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.695987    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:03.696116 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.696104    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.696161 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.696131    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:03.696704 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.696689    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:03.696774 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.696711    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:03.696774 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.696721    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:03.696774 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.696736    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:03.696774 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.696755    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:03.696774 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.696769    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:03.697779 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.697766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.697829 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.697790    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 17:17:03.698449 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.698434    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 17:17:03.698521 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.698462    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 17:17:03.698521 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.698473    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeHasSufficientPID"
Jul 28 17:17:03.713210 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.713189    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-64.ec2.internal\" not found" node="ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.717034 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.717020    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-137-64.ec2.internal\" not found" node="ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.731102 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.731083    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:03.740541 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.740518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f56741a033333d3b3c44b4959f345dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal\" (UID: \"7f56741a033333d3b3c44b4959f345dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.740612 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.740550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f56741a033333d3b3c44b4959f345dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal\" (UID: \"7f56741a033333d3b3c44b4959f345dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.740612 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.740575    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2557c4e520af9150ace36fc6434afd7f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-64.ec2.internal\" (UID: \"2557c4e520af9150ace36fc6434afd7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.831138 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.831119    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:03.841536 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.841515    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2557c4e520af9150ace36fc6434afd7f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-64.ec2.internal\" (UID: \"2557c4e520af9150ace36fc6434afd7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.841606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.841544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f56741a033333d3b3c44b4959f345dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal\" (UID: \"7f56741a033333d3b3c44b4959f345dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.841606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.841570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f56741a033333d3b3c44b4959f345dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal\" (UID: \"7f56741a033333d3b3c44b4959f345dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.841715 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.841611    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2557c4e520af9150ace36fc6434afd7f-config\") pod \"kube-apiserver-proxy-ip-10-0-137-64.ec2.internal\" (UID: \"2557c4e520af9150ace36fc6434afd7f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.841715 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.841616    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7f56741a033333d3b3c44b4959f345dd-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal\" (UID: \"7f56741a033333d3b3c44b4959f345dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.841715 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:03.841622    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7f56741a033333d3b3c44b4959f345dd-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal\" (UID: \"7f56741a033333d3b3c44b4959f345dd\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:03.931512 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:03.931492    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:04.020832 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.020786    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:04.021977 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.021955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:04.032057 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:04.032037    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:04.132571 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:04.132489    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:04.235065 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:04.235028    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:04.257127 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.257107    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:04.338601 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:04.338574    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-137-64.ec2.internal\" not found"
Jul 28 17:17:04.372340 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.372318    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 17:17:04.372758 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.372450    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:17:04.372758 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.372450    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 17:17:04.415772 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.415720    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:04.438139 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.438118    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:04.438230 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.438127    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 17:17:04.449168 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.449139    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 17:12:03 +0000 UTC" deadline="2028-04-24 09:13:04.545424356 +0000 UTC"
Jul 28 17:17:04.449168 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.449162    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15255h56m0.09626467s"
Jul 28 17:17:04.450378 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.450365    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:17:04.452465 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.452453    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal"
Jul 28 17:17:04.468073 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.468051    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 17:17:04.469767 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.469739    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 17:17:04.479986 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.479966    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p4z8f"
Jul 28 17:17:04.495214 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.495192    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p4z8f"
Jul 28 17:17:04.626091 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.626063    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:04.726890 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:04.726858    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f56741a033333d3b3c44b4959f345dd.slice/crio-1dd16affcfc25e0bed73720b08cf794ea46eab63330e32d03d60e43aa8b38805 WatchSource:0}: Error finding container 1dd16affcfc25e0bed73720b08cf794ea46eab63330e32d03d60e43aa8b38805: Status 404 returned error can't find the container with id 1dd16affcfc25e0bed73720b08cf794ea46eab63330e32d03d60e43aa8b38805
Jul 28 17:17:04.727388 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:04.727370    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2557c4e520af9150ace36fc6434afd7f.slice/crio-1b19fdc9431f942f9a857d17508e99835b87a6d26b92210c178f754b4d2959c0 WatchSource:0}: Error finding container 1b19fdc9431f942f9a857d17508e99835b87a6d26b92210c178f754b4d2959c0: Status 404 returned error can't find the container with id 1b19fdc9431f942f9a857d17508e99835b87a6d26b92210c178f754b4d2959c0
Jul 28 17:17:04.733474 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:04.733455    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:17:05.355267 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.355235    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:05.415044 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.415016    2563 apiserver.go:52] "Watching apiserver"
Jul 28 17:17:05.425581 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.425556    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 17:17:05.427151 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.426728    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-sgtxd","openshift-dns/node-resolver-qh26v","openshift-image-registry/node-ca-6b6qm","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal","openshift-network-diagnostics/network-check-target-mlbvr","openshift-network-operator/iptables-alerter-z4k8t","kube-system/global-pull-secret-syncer-gjl4f","kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j","openshift-multus/multus-additional-cni-plugins-dtzv7","openshift-multus/multus-wh264","openshift-multus/network-metrics-daemon-rlbl4","openshift-ovn-kubernetes/ovnkube-node-56rvp","kube-system/konnectivity-agent-cn5vl"]
Jul 28 17:17:05.428424 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.428399    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.429687 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.429651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.431557 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.430821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.431866 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.431845    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:05.432159 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.432139    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-xp4xw\""
Jul 28 17:17:05.432477 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.432459    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:05.433070 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.433050    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 17:17:05.434096 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.434076    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:05.434205 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.434179    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:05.434605 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.434582    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 17:17:05.434768 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.434718    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 17:17:05.434899 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.434881    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 17:17:05.434979 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.434962    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 17:17:05.435035 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.435009    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-ptng8\""
Jul 28 17:17:05.435437 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.435416    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-r4jfd\""
Jul 28 17:17:05.436725 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.436344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.437641 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.437516    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.437641 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.437601    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:05.438782 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.438750    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.440939 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.440324    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 17:17:05.440939 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.440680    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 17:17:05.441048 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.440986    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:05.441048 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.441017    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-td5dk\""
Jul 28 17:17:05.441463 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.441447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.443357 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.443219    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 17:17:05.443493 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.443476    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 17:17:05.446673 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.444769    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.446673 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.444926    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 17:17:05.446673 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.445167    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 17:17:05.446673 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.445577    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 17:17:05.446673 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.445968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-ksdjw\""
Jul 28 17:17:05.446673 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.446095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:05.446673 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.446183    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:05.446673 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.446361    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 17:17:05.447117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.446932    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 17:17:05.449081 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.448828    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6k2l\""
Jul 28 17:17:05.449081 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.448954    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 17:17:05.449286 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.449178    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.450613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.450561    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 17:17:05.450613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.450588    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-6h7zl\""
Jul 28 17:17:05.451085 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzg89\" (UniqueName: \"kubernetes.io/projected/ccbf21fa-84d8-4348-bc89-36ea89e9284f-kube-api-access-lzg89\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451542    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-sys\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccbf21fa-84d8-4348-bc89-36ea89e9284f-host\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysctl-d\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79946742-16f6-4416-a9d2-e54871e8ed2d-iptables-alerter-script\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451655    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-tmp-dir\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/42b3b5e9-0f43-4c50-be37-a7c61ff99746-kubelet-config\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-etc-selinux\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rrgt\" (UniqueName: \"kubernetes.io/projected/79946742-16f6-4416-a9d2-e54871e8ed2d-kube-api-access-2rrgt\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-hosts-file\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451829    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysctl-conf\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-systemd\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451899    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-lib-modules\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451927    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ccbf21fa-84d8-4348-bc89-36ea89e9284f-serviceca\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451953    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-sys-fs\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-modprobe-d\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.451999    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-socket-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysconfig\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-tuned\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452088    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v45r9\" (UniqueName: \"kubernetes.io/projected/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-kube-api-access-v45r9\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452131    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79946742-16f6-4416-a9d2-e54871e8ed2d-host-slash\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452184    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-device-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-run\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452231    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-var-lib-kubelet\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452253    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-host\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-tmp\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452300    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zz5gs\" (UniqueName: \"kubernetes.io/projected/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-kube-api-access-zz5gs\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452323    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/42b3b5e9-0f43-4c50-be37-a7c61ff99746-dbus\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.452946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-registration-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.453862 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc2nm\" (UniqueName: \"kubernetes.io/projected/8e2c27f5-8271-4f35-b2d1-b34649409743-kube-api-access-wc2nm\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.453862 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.452401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-kubernetes\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.453862 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.453553    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 17:17:05.453862 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.453597    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 17:17:05.453862 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.453554    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-7q5gm\""
Jul 28 17:17:05.455202 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.455182    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 17:17:05.456336 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.455408    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 17:17:05.456336 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.455637    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 17:17:05.456336 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.455701    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 17:17:05.456336 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.455979    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 17:17:05.456336 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.455990    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-7rxd8\""
Jul 28 17:17:05.456336 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.456198    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 17:17:05.497108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.497044    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:12:04 +0000 UTC" deadline="2028-02-10 11:50:44.956539902 +0000 UTC"
Jul 28 17:17:05.497108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.497105    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13482h33m39.459438993s"
Jul 28 17:17:05.538936 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.538915    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 17:17:05.552979 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.552951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-node-log\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553082 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.552991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.553082 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-systemd\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553082 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553214 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovn-node-metrics-cert\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553214 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-tmp\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.553214 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-os-release\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.553214 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553198    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.553424 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553228    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-system-cni-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.553424 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-etc-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553424 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553400    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/767e90f1-3aaa-4ec1-adb0-e732020844d5-konnectivity-ca\") pod \"konnectivity-agent-cn5vl\" (UID: \"767e90f1-3aaa-4ec1-adb0-e732020844d5\") " pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:05.553573 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zz5gs\" (UniqueName: \"kubernetes.io/projected/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-kube-api-access-zz5gs\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.553573 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-registration-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.553573 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553502    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 17:17:05.553573 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-kubernetes\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.553573 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553561    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-hostroot\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.553841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553588    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-conf-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.553841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553636    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-var-lib-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovnkube-config\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553689    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccbf21fa-84d8-4348-bc89-36ea89e9284f-host\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.553841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cnibin\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.553841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553780    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-run-netns\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-ovn\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.553841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysctl-d\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.554157 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.553857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-netns\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.554157 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554041    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysctl-d\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.554157 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554079    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovnkube-script-lib\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.554157 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-tmp-dir\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.554157 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/42b3b5e9-0f43-4c50-be37-a7c61ff99746-kubelet-config\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554164    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-system-cni-dir\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-kubelet\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rrgt\" (UniqueName: \"kubernetes.io/projected/79946742-16f6-4416-a9d2-e54871e8ed2d-kube-api-access-2rrgt\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-hosts-file\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554266    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-lib-modules\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ccbf21fa-84d8-4348-bc89-36ea89e9284f-serviceca\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/767e90f1-3aaa-4ec1-adb0-e732020844d5-agent-certs\") pod \"konnectivity-agent-cn5vl\" (UID: \"767e90f1-3aaa-4ec1-adb0-e732020844d5\") " pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554354    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35770e90-7d87-4644-918f-aaea48f2e530-multus-daemon-config\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.554388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ccbf21fa-84d8-4348-bc89-36ea89e9284f-host\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/42b3b5e9-0f43-4c50-be37-a7c61ff99746-kubelet-config\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554470    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhcxt\" (UniqueName: \"kubernetes.io/projected/26a3cd7c-de8f-4208-b600-b598dd82afd4-kube-api-access-vhcxt\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554499    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-slash\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554533    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-lib-modules\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-log-socket\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554483    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-hosts-file\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-socket-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysconfig\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554670    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-tuned\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v45r9\" (UniqueName: \"kubernetes.io/projected/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-kube-api-access-v45r9\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554724    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-socket-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554751    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-tmp-dir\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554771    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554784    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-registration-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-kubernetes\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.554798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554807    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-systemd-units\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-kubelet\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.554887    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79946742-16f6-4416-a9d2-e54871e8ed2d-host-slash\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555073    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/79946742-16f6-4416-a9d2-e54871e8ed2d-host-slash\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-device-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555127    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysconfig\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555168    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-run\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ccbf21fa-84d8-4348-bc89-36ea89e9284f-serviceca\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555218    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-device-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-var-lib-kubelet\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555271    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-host\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-var-lib-kubelet\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/42b3b5e9-0f43-4c50-be37-a7c61ff99746-dbus\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555327    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-host\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555329    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-run\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc2nm\" (UniqueName: \"kubernetes.io/projected/8e2c27f5-8271-4f35-b2d1-b34649409743-kube-api-access-wc2nm\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.555627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/42b3b5e9-0f43-4c50-be37-a7c61ff99746-dbus\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555884    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzg89\" (UniqueName: \"kubernetes.io/projected/ccbf21fa-84d8-4348-bc89-36ea89e9284f-kube-api-access-lzg89\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555922    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35770e90-7d87-4644-918f-aaea48f2e530-cni-binary-copy\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrbcj\" (UniqueName: \"kubernetes.io/projected/a728637b-8a7c-4a0a-bb2d-b9b18c258162-kube-api-access-qrbcj\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.555980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-socket-dir-parent\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-cni-multus\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-sys\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556134    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-os-release\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-sys\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-cni-netd\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556211    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-cni-bin\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-multus-certs\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-etc-kubernetes\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cglbg\" (UniqueName: \"kubernetes.io/projected/35770e90-7d87-4644-918f-aaea48f2e530-kube-api-access-cglbg\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-cni-bin\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.556431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556396    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdnf4\" (UniqueName: \"kubernetes.io/projected/17c64fb3-f111-4c84-9b47-6c4695055d2f-kube-api-access-zdnf4\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79946742-16f6-4416-a9d2-e54871e8ed2d-iptables-alerter-script\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-etc-selinux\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-etc-selinux\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556643    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-kubelet-dir\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556725    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysctl-conf\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-systemd\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.556826    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-systemd\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556925    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-sysctl-conf\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.556936    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret podName:42b3b5e9-0f43-4c50-be37-a7c61ff99746 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:06.056899466 +0000 UTC m=+2.990298885 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret") pod "global-pull-secret-syncer-gjl4f" (UID: "42b3b5e9-0f43-4c50-be37-a7c61ff99746") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.556978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-cni-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557020    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-cnibin\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-env-overrides\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.557264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-sys-fs\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.557993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-modprobe-d\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.557993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557126    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-k8s-cni-cncf-io\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.557993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/79946742-16f6-4416-a9d2-e54871e8ed2d-iptables-alerter-script\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.557993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/8e2c27f5-8271-4f35-b2d1-b34649409743-sys-fs\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.557993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-modprobe-d\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.557993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557583    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-tmp\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.557993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.557604    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-etc-tuned\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.565318 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.565295    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:05.565431 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.565320    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:05.565431 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.565341    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfxmm for pod openshift-network-diagnostics/network-check-target-mlbvr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:05.565431 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.565404    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm podName:94714598-e34a-4953-b07d-c1b9037a561e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:06.065386029 +0000 UTC m=+2.998785427 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-cfxmm" (UniqueName: "kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm") pod "network-check-target-mlbvr" (UID: "94714598-e34a-4953-b07d-c1b9037a561e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:05.569951 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.569590    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zz5gs\" (UniqueName: \"kubernetes.io/projected/9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c-kube-api-access-zz5gs\") pod \"node-resolver-qh26v\" (UID: \"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c\") " pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.570050 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.569953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rrgt\" (UniqueName: \"kubernetes.io/projected/79946742-16f6-4416-a9d2-e54871e8ed2d-kube-api-access-2rrgt\") pod \"iptables-alerter-z4k8t\" (UID: \"79946742-16f6-4416-a9d2-e54871e8ed2d\") " pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.571623 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.571595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v45r9\" (UniqueName: \"kubernetes.io/projected/57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4-kube-api-access-v45r9\") pod \"tuned-sgtxd\" (UID: \"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4\") " pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.574610 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.574580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc2nm\" (UniqueName: \"kubernetes.io/projected/8e2c27f5-8271-4f35-b2d1-b34649409743-kube-api-access-wc2nm\") pod \"aws-ebs-csi-driver-node-h944j\" (UID: \"8e2c27f5-8271-4f35-b2d1-b34649409743\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.575754 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.575735    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzg89\" (UniqueName: \"kubernetes.io/projected/ccbf21fa-84d8-4348-bc89-36ea89e9284f-kube-api-access-lzg89\") pod \"node-ca-6b6qm\" (UID: \"ccbf21fa-84d8-4348-bc89-36ea89e9284f\") " pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.595906 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.595858    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal" event={"ID":"2557c4e520af9150ace36fc6434afd7f","Type":"ContainerStarted","Data":"1b19fdc9431f942f9a857d17508e99835b87a6d26b92210c178f754b4d2959c0"}
Jul 28 17:17:05.597425 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.597398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal" event={"ID":"7f56741a033333d3b3c44b4959f345dd","Type":"ContainerStarted","Data":"1dd16affcfc25e0bed73720b08cf794ea46eab63330e32d03d60e43aa8b38805"}
Jul 28 17:17:05.658148 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-system-cni-dir\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.658148 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-kubelet\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.658148 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658138    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-system-cni-dir\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.658406 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/767e90f1-3aaa-4ec1-adb0-e732020844d5-agent-certs\") pod \"konnectivity-agent-cn5vl\" (UID: \"767e90f1-3aaa-4ec1-adb0-e732020844d5\") " pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:05.658406 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658206    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-kubelet\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.658406 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35770e90-7d87-4644-918f-aaea48f2e530-multus-daemon-config\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.658406 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658345    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhcxt\" (UniqueName: \"kubernetes.io/projected/26a3cd7c-de8f-4208-b600-b598dd82afd4-kube-api-access-vhcxt\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:05.658406 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-slash\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658439    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-log-socket\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658470    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658494    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-systemd-units\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658519    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-kubelet\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658566    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35770e90-7d87-4644-918f-aaea48f2e530-cni-binary-copy\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658591    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrbcj\" (UniqueName: \"kubernetes.io/projected/a728637b-8a7c-4a0a-bb2d-b9b18c258162-kube-api-access-qrbcj\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-socket-dir-parent\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.658659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658641    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-log-socket\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658644    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-cni-multus\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658684    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-cni-multus\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658748    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-os-release\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-cni-netd\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658802    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-cni-bin\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658826    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/35770e90-7d87-4644-918f-aaea48f2e530-multus-daemon-config\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658829    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-multus-certs\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658862    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-multus-certs\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-tuning-conf-dir\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-etc-kubernetes\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658953    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-systemd-units\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.659019    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:05.659149 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:05.659082    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs podName:26a3cd7c-de8f-4208-b600-b598dd82afd4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:06.159064903 +0000 UTC m=+3.092464300 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs") pod "network-metrics-daemon-rlbl4" (UID: "26a3cd7c-de8f-4208-b600-b598dd82afd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-socket-dir-parent\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659234    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-os-release\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-slash\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-cni-netd\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-cni-bin\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-var-lib-kubelet\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.658875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-etc-kubernetes\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cglbg\" (UniqueName: \"kubernetes.io/projected/35770e90-7d87-4644-918f-aaea48f2e530-kube-api-access-cglbg\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659415    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-cni-bin\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdnf4\" (UniqueName: \"kubernetes.io/projected/17c64fb3-f111-4c84-9b47-6c4695055d2f-kube-api-access-zdnf4\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659522    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-run-ovn-kubernetes\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659584    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-cni-bin\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-cni-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659714    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-cnibin\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.659874 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659739    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-env-overrides\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659787    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-k8s-cni-cncf-io\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-node-log\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/35770e90-7d87-4644-918f-aaea48f2e530-cni-binary-copy\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-node-log\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-cnibin\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-cni-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.659987    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-k8s-cni-cncf-io\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-systemd\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovn-node-metrics-cert\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-os-release\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-env-overrides\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-system-cni-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-etc-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.660637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660458    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/767e90f1-3aaa-4ec1-adb0-e732020844d5-konnectivity-ca\") pod \"konnectivity-agent-cn5vl\" (UID: \"767e90f1-3aaa-4ec1-adb0-e732020844d5\") " pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660486    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-hostroot\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-conf-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-var-lib-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660563    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovnkube-config\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cnibin\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660615    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-run-netns\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660641    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-ovn\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-netns\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovnkube-script-lib\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-host-run-netns\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660810    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-system-cni-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-etc-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.660933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovnkube-script-lib\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.661497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661345    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/767e90f1-3aaa-4ec1-adb0-e732020844d5-konnectivity-ca\") pod \"konnectivity-agent-cn5vl\" (UID: \"767e90f1-3aaa-4ec1-adb0-e732020844d5\") " pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-hostroot\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661461    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/35770e90-7d87-4644-918f-aaea48f2e530-multus-conf-dir\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-var-lib-openvswitch\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/767e90f1-3aaa-4ec1-adb0-e732020844d5-agent-certs\") pod \"konnectivity-agent-cn5vl\" (UID: \"767e90f1-3aaa-4ec1-adb0-e732020844d5\") " pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661658    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-systemd\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-run-ovn\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661815    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/17c64fb3-f111-4c84-9b47-6c4695055d2f-host-run-netns\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661844    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cnibin\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a728637b-8a7c-4a0a-bb2d-b9b18c258162-os-release\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.661981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovnkube-config\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.662469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.662194    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a728637b-8a7c-4a0a-bb2d-b9b18c258162-cni-binary-copy\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.663988 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.663944    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/17c64fb3-f111-4c84-9b47-6c4695055d2f-ovn-node-metrics-cert\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.670500 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.670459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrbcj\" (UniqueName: \"kubernetes.io/projected/a728637b-8a7c-4a0a-bb2d-b9b18c258162-kube-api-access-qrbcj\") pod \"multus-additional-cni-plugins-dtzv7\" (UID: \"a728637b-8a7c-4a0a-bb2d-b9b18c258162\") " pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.671778 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.671754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhcxt\" (UniqueName: \"kubernetes.io/projected/26a3cd7c-de8f-4208-b600-b598dd82afd4-kube-api-access-vhcxt\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:05.675482 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.675433    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdnf4\" (UniqueName: \"kubernetes.io/projected/17c64fb3-f111-4c84-9b47-6c4695055d2f-kube-api-access-zdnf4\") pod \"ovnkube-node-56rvp\" (UID: \"17c64fb3-f111-4c84-9b47-6c4695055d2f\") " pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.677163 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.677144    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cglbg\" (UniqueName: \"kubernetes.io/projected/35770e90-7d87-4644-918f-aaea48f2e530-kube-api-access-cglbg\") pod \"multus-wh264\" (UID: \"35770e90-7d87-4644-918f-aaea48f2e530\") " pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.744940 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.744909    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-sgtxd"
Jul 28 17:17:05.756134 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.756113    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qh26v"
Jul 28 17:17:05.767993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.767968    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6b6qm"
Jul 28 17:17:05.775784 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.775766    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-z4k8t"
Jul 28 17:17:05.785375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.785357    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j"
Jul 28 17:17:05.793049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.793024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-dtzv7"
Jul 28 17:17:05.801246 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.801228    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-wh264"
Jul 28 17:17:05.808971 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.808952    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:05.816054 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:05.816034    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:06.065399 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:06.064593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:06.065399 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.064799    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:06.065399 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.064869    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret podName:42b3b5e9-0f43-4c50-be37-a7c61ff99746 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:07.064848791 +0000 UTC m=+3.998248205 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret") pod "global-pull-secret-syncer-gjl4f" (UID: "42b3b5e9-0f43-4c50-be37-a7c61ff99746") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:06.165553 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:06.165517    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:06.165716 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:06.165562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:06.165716 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.165695    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:06.165824 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.165716    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:06.165824 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.165736    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:06.165824 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.165751    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfxmm for pod openshift-network-diagnostics/network-check-target-mlbvr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:06.165824 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.165768    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs podName:26a3cd7c-de8f-4208-b600-b598dd82afd4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:07.165748005 +0000 UTC m=+4.099147402 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs") pod "network-metrics-daemon-rlbl4" (UID: "26a3cd7c-de8f-4208-b600-b598dd82afd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:06.165824 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.165795    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm podName:94714598-e34a-4953-b07d-c1b9037a561e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:07.165782704 +0000 UTC m=+4.099182102 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfxmm" (UniqueName: "kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm") pod "network-check-target-mlbvr" (UID: "94714598-e34a-4953-b07d-c1b9037a561e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:06.321923 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:06.321842    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 17:17:06.498013 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:06.497963    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 17:12:04 +0000 UTC" deadline="2028-01-04 08:08:21.292949089 +0000 UTC"
Jul 28 17:17:06.498013 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:06.498006    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12590h51m14.794945688s"
Jul 28 17:17:06.591196 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:06.591128    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:06.591352 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:06.591243    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:06.809087 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:06.809055    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35770e90_7d87_4644_918f_aaea48f2e530.slice/crio-7eae86061bd368b2969b1a8b3fa2a2fd0e0c77b4750f224d43f52422a2b3fb49 WatchSource:0}: Error finding container 7eae86061bd368b2969b1a8b3fa2a2fd0e0c77b4750f224d43f52422a2b3fb49: Status 404 returned error can't find the container with id 7eae86061bd368b2969b1a8b3fa2a2fd0e0c77b4750f224d43f52422a2b3fb49
Jul 28 17:17:06.811440 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:06.811348    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79946742_16f6_4416_a9d2_e54871e8ed2d.slice/crio-f331ef3136c74e33f957f94b1de76809ffc7bfdc7bc5ae0d02ab75615107e716 WatchSource:0}: Error finding container f331ef3136c74e33f957f94b1de76809ffc7bfdc7bc5ae0d02ab75615107e716: Status 404 returned error can't find the container with id f331ef3136c74e33f957f94b1de76809ffc7bfdc7bc5ae0d02ab75615107e716
Jul 28 17:17:06.813942 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:06.813340    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b96ddc0_b47c_4e45_989d_fcd1d1cd2e8c.slice/crio-40a7af98b36bbb2061300d3ab7333fcdb4831bd5929c16e773cf59d714c65982 WatchSource:0}: Error finding container 40a7af98b36bbb2061300d3ab7333fcdb4831bd5929c16e773cf59d714c65982: Status 404 returned error can't find the container with id 40a7af98b36bbb2061300d3ab7333fcdb4831bd5929c16e773cf59d714c65982
Jul 28 17:17:06.815006 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:06.814887    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2c27f5_8271_4f35_b2d1_b34649409743.slice/crio-58f7d169399e2b32a8f86caa7281970c02a26890e9c0d41ae6a20ec7c3449847 WatchSource:0}: Error finding container 58f7d169399e2b32a8f86caa7281970c02a26890e9c0d41ae6a20ec7c3449847: Status 404 returned error can't find the container with id 58f7d169399e2b32a8f86caa7281970c02a26890e9c0d41ae6a20ec7c3449847
Jul 28 17:17:06.815906 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:06.815787    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57fa9c43_75d8_4b1b_91cc_b4fe41d43dc4.slice/crio-c1fe70f0acbe675594f3188f3f069ffeeca0469da907382b4c814232c3837bdf WatchSource:0}: Error finding container c1fe70f0acbe675594f3188f3f069ffeeca0469da907382b4c814232c3837bdf: Status 404 returned error can't find the container with id c1fe70f0acbe675594f3188f3f069ffeeca0469da907382b4c814232c3837bdf
Jul 28 17:17:06.817377 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:06.817012    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17c64fb3_f111_4c84_9b47_6c4695055d2f.slice/crio-62cd52be03e17f275aa6d731c59bbc55d7cf460d5d4f94cbb19205e004133465 WatchSource:0}: Error finding container 62cd52be03e17f275aa6d731c59bbc55d7cf460d5d4f94cbb19205e004133465: Status 404 returned error can't find the container with id 62cd52be03e17f275aa6d731c59bbc55d7cf460d5d4f94cbb19205e004133465
Jul 28 17:17:06.818241 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:06.818124    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podccbf21fa_84d8_4348_bc89_36ea89e9284f.slice/crio-0f241e88a9d62519304702898fc5f1e4d3b0703968db95945149ae84a12bbdf6 WatchSource:0}: Error finding container 0f241e88a9d62519304702898fc5f1e4d3b0703968db95945149ae84a12bbdf6: Status 404 returned error can't find the container with id 0f241e88a9d62519304702898fc5f1e4d3b0703968db95945149ae84a12bbdf6
Jul 28 17:17:07.074319 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.074136    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:07.074404 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.074279    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:07.074404 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.074390    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret podName:42b3b5e9-0f43-4c50-be37-a7c61ff99746 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:09.074375148 +0000 UTC m=+6.007774546 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret") pod "global-pull-secret-syncer-gjl4f" (UID: "42b3b5e9-0f43-4c50-be37-a7c61ff99746") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:07.174996 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.174913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:07.174996 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.174954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:07.175158 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.175066    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:07.175158 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.175117    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:07.175158 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.175126    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs podName:26a3cd7c-de8f-4208-b600-b598dd82afd4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:09.175106343 +0000 UTC m=+6.108505753 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs") pod "network-metrics-daemon-rlbl4" (UID: "26a3cd7c-de8f-4208-b600-b598dd82afd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:07.175158 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.175137    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:07.175158 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.175151    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfxmm for pod openshift-network-diagnostics/network-check-target-mlbvr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:07.175345 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.175199    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm podName:94714598-e34a-4953-b07d-c1b9037a561e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:09.175183246 +0000 UTC m=+6.108582665 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfxmm" (UniqueName: "kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm") pod "network-check-target-mlbvr" (UID: "94714598-e34a-4953-b07d-c1b9037a561e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:07.595738 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.593416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:07.595738 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.593567    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:07.595738 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.594008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:07.595738 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:07.594102    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:07.610838 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.610803    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cn5vl" event={"ID":"767e90f1-3aaa-4ec1-adb0-e732020844d5","Type":"ContainerStarted","Data":"125aa6983e7985b8722c0bf1b4b9f95e2b41931ac4c26afee38f9957561b75ef"}
Jul 28 17:17:07.613403 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.613338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wh264" event={"ID":"35770e90-7d87-4644-918f-aaea48f2e530","Type":"ContainerStarted","Data":"7eae86061bd368b2969b1a8b3fa2a2fd0e0c77b4750f224d43f52422a2b3fb49"}
Jul 28 17:17:07.619879 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.619258    2563 generic.go:356] "Generic (PLEG): container finished" podID="7f56741a033333d3b3c44b4959f345dd" containerID="cfc5af7c99e78252499f4d2f6f6ab78899bb118f4a6794e7deabf19c16b919c9" exitCode=0
Jul 28 17:17:07.619879 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.619347    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal" event={"ID":"7f56741a033333d3b3c44b4959f345dd","Type":"ContainerDied","Data":"cfc5af7c99e78252499f4d2f6f6ab78899bb118f4a6794e7deabf19c16b919c9"}
Jul 28 17:17:07.623804 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.623734    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6b6qm" event={"ID":"ccbf21fa-84d8-4348-bc89-36ea89e9284f","Type":"ContainerStarted","Data":"0f241e88a9d62519304702898fc5f1e4d3b0703968db95945149ae84a12bbdf6"}
Jul 28 17:17:07.635850 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.635819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"62cd52be03e17f275aa6d731c59bbc55d7cf460d5d4f94cbb19205e004133465"}
Jul 28 17:17:07.641455 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.641387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sgtxd" event={"ID":"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4","Type":"ContainerStarted","Data":"c1fe70f0acbe675594f3188f3f069ffeeca0469da907382b4c814232c3837bdf"}
Jul 28 17:17:07.644053 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.644025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j" event={"ID":"8e2c27f5-8271-4f35-b2d1-b34649409743","Type":"ContainerStarted","Data":"58f7d169399e2b32a8f86caa7281970c02a26890e9c0d41ae6a20ec7c3449847"}
Jul 28 17:17:07.666311 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.666224    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z4k8t" event={"ID":"79946742-16f6-4416-a9d2-e54871e8ed2d","Type":"ContainerStarted","Data":"f331ef3136c74e33f957f94b1de76809ffc7bfdc7bc5ae0d02ab75615107e716"}
Jul 28 17:17:07.680762 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.680718    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qh26v" event={"ID":"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c","Type":"ContainerStarted","Data":"40a7af98b36bbb2061300d3ab7333fcdb4831bd5929c16e773cf59d714c65982"}
Jul 28 17:17:07.694585 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.694558    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" event={"ID":"a728637b-8a7c-4a0a-bb2d-b9b18c258162","Type":"ContainerStarted","Data":"c81a4cf41a7328d8e4bdc99adc389a6332be6ad95a6cf346a9bee5395e698b7d"}
Jul 28 17:17:07.700218 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:07.700189    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal" event={"ID":"2557c4e520af9150ace36fc6434afd7f","Type":"ContainerStarted","Data":"c759b94df13176e1e8da01541db6a1c837114078ca281d3e3ad5537bcb4ebf04"}
Jul 28 17:17:08.591212 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:08.591075    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:08.591434 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:08.591218    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:08.727124 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:08.726931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal" event={"ID":"7f56741a033333d3b3c44b4959f345dd","Type":"ContainerStarted","Data":"0fff13e476b91029f0ec40198db4b93e22f560d5e29e7a72705157b1793e6264"}
Jul 28 17:17:08.747761 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:08.747697    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-137-64.ec2.internal" podStartSLOduration=4.747679061 podStartE2EDuration="4.747679061s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:07.71965041 +0000 UTC m=+4.653049836" watchObservedRunningTime="2026-07-28 17:17:08.747679061 +0000 UTC m=+5.681078485"
Jul 28 17:17:08.748679 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:08.748632    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-137-64.ec2.internal" podStartSLOduration=4.748618357 podStartE2EDuration="4.748618357s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:08.747595349 +0000 UTC m=+5.680994775" watchObservedRunningTime="2026-07-28 17:17:08.748618357 +0000 UTC m=+5.682017782"
Jul 28 17:17:09.095243 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:09.095203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:09.095433 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.095363    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:09.095433 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.095430    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret podName:42b3b5e9-0f43-4c50-be37-a7c61ff99746 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:13.095409988 +0000 UTC m=+10.028809392 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret") pod "global-pull-secret-syncer-gjl4f" (UID: "42b3b5e9-0f43-4c50-be37-a7c61ff99746") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:09.195840 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:09.195803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:09.196023 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:09.195860    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:09.196092 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.196079    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:09.196151 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.196099    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:09.196151 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.196114    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfxmm for pod openshift-network-diagnostics/network-check-target-mlbvr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:09.196253 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.196175    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm podName:94714598-e34a-4953-b07d-c1b9037a561e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:13.196154952 +0000 UTC m=+10.129554353 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfxmm" (UniqueName: "kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm") pod "network-check-target-mlbvr" (UID: "94714598-e34a-4953-b07d-c1b9037a561e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:09.196601 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.196580    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:09.196706 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.196637    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs podName:26a3cd7c-de8f-4208-b600-b598dd82afd4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:13.196621219 +0000 UTC m=+10.130020618 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs") pod "network-metrics-daemon-rlbl4" (UID: "26a3cd7c-de8f-4208-b600-b598dd82afd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:09.592955 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:09.592877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:09.593113 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.593031    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:09.593484 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:09.593460    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:09.593580 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:09.593561    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:10.591652 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:10.591185    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:10.591652 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:10.591321    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:11.591594 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:11.591564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:11.591811 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:11.591702    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:11.592194 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:11.592130    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:11.592295 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:11.592239    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:12.591829 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:12.591791    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:12.592243 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:12.591930    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:13.128746 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:13.128709    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:13.128935 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.128864    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:13.128935 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.128934    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret podName:42b3b5e9-0f43-4c50-be37-a7c61ff99746 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:21.128913619 +0000 UTC m=+18.062313029 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret") pod "global-pull-secret-syncer-gjl4f" (UID: "42b3b5e9-0f43-4c50-be37-a7c61ff99746") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:13.230690 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:13.230001    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:13.230690 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:13.230052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:13.230690 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.230242    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:13.230690 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.230260    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:13.230690 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.230273    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfxmm for pod openshift-network-diagnostics/network-check-target-mlbvr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:13.230690 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.230328    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm podName:94714598-e34a-4953-b07d-c1b9037a561e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:21.230310332 +0000 UTC m=+18.163709727 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfxmm" (UniqueName: "kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm") pod "network-check-target-mlbvr" (UID: "94714598-e34a-4953-b07d-c1b9037a561e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:13.230690 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.230588    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:13.230690 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.230636    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs podName:26a3cd7c-de8f-4208-b600-b598dd82afd4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:21.230620384 +0000 UTC m=+18.164019784 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs") pod "network-metrics-daemon-rlbl4" (UID: "26a3cd7c-de8f-4208-b600-b598dd82afd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:13.592454 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:13.591926    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:13.592454 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.592040    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:13.592454 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:13.592090    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:13.592454 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:13.592208    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:14.591498 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:14.591462    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:14.591691 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:14.591614    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:15.591917 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:15.591872    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:15.592438 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:15.592009    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:15.592438 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:15.592063    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:15.592438 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:15.592178    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:16.591745 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:16.591713    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:16.591912 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:16.591841    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:17.591884 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:17.591850    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:17.591884 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:17.591874    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:17.592323 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:17.591972    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:17.592323 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:17.592100    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:18.590953 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:18.590918    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:18.591158 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:18.591046    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:19.591093 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:19.591057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:19.591473 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:19.591195    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:19.591473 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:19.591249    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:19.591473 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:19.591368    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:20.591776 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:20.591744    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:20.592192 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:20.591850    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:21.189864 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:21.189826    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:21.190011 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.189957    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:21.190064 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.190015    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret podName:42b3b5e9-0f43-4c50-be37-a7c61ff99746 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:37.189999741 +0000 UTC m=+34.123399136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret") pod "global-pull-secret-syncer-gjl4f" (UID: "42b3b5e9-0f43-4c50-be37-a7c61ff99746") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:21.290330 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:21.290291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:21.290330 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:21.290328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:21.290549 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.290443    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:21.290549 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.290455    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:21.290549 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.290469    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:21.290549 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.290482    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfxmm for pod openshift-network-diagnostics/network-check-target-mlbvr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:21.290549 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.290504    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs podName:26a3cd7c-de8f-4208-b600-b598dd82afd4 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:37.290484929 +0000 UTC m=+34.223884345 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs") pod "network-metrics-daemon-rlbl4" (UID: "26a3cd7c-de8f-4208-b600-b598dd82afd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:21.290549 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.290531    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm podName:94714598-e34a-4953-b07d-c1b9037a561e nodeName:}" failed. No retries permitted until 2026-07-28 17:17:37.290516123 +0000 UTC m=+34.223915542 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfxmm" (UniqueName: "kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm") pod "network-check-target-mlbvr" (UID: "94714598-e34a-4953-b07d-c1b9037a561e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:21.591782 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:21.591712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:21.591782 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:21.591728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:21.592264 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.591858    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:21.592264 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:21.591972    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:22.591761 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:22.591726    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:22.591965 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:22.591856    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:23.591948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:23.591916    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:23.592120 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:23.592015    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:23.592451 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:23.592347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:23.592451 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:23.592426    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:24.591177 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:24.591149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:24.591343 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:24.591245    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:25.591196 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.591174    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:25.591547 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.591192    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:25.591547 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:25.591394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:25.591547 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:25.591488    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:25.760760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.760498    2563 generic.go:356] "Generic (PLEG): container finished" podID="a728637b-8a7c-4a0a-bb2d-b9b18c258162" containerID="1ddf0f108de5e31784e05a293a527af54bf607e969d5be811bf1cef4060f337f" exitCode=0
Jul 28 17:17:25.760875 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.760596    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" event={"ID":"a728637b-8a7c-4a0a-bb2d-b9b18c258162","Type":"ContainerDied","Data":"1ddf0f108de5e31784e05a293a527af54bf607e969d5be811bf1cef4060f337f"}
Jul 28 17:17:25.762242 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.762213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cn5vl" event={"ID":"767e90f1-3aaa-4ec1-adb0-e732020844d5","Type":"ContainerStarted","Data":"1a09de0864700cb2d9a1ff0c33e7bfad6219d66a46f091dc93a449b6dcc6859a"}
Jul 28 17:17:25.763937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.763911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-wh264" event={"ID":"35770e90-7d87-4644-918f-aaea48f2e530","Type":"ContainerStarted","Data":"94922142980bfcfe72b9e89c4f6c9a702b5580e3c5bd4984d6da1b4680e5614b"}
Jul 28 17:17:25.765540 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.765517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6b6qm" event={"ID":"ccbf21fa-84d8-4348-bc89-36ea89e9284f","Type":"ContainerStarted","Data":"00dbc70de08380b5e325ef068d38aca6e50c32519cfb13bc73d25dee1a47203c"}
Jul 28 17:17:25.768284 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.768250    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"87869655e82ccdc1272d6041b2e630b6301b240be4efb81ec86b63ca408fe896"}
Jul 28 17:17:25.768357 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.768320    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"bd8196a3ea8869b6ba2c6c7393e3352c8f7d3546c88a76b7e1dc0c1e2712701f"}
Jul 28 17:17:25.770029 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.769862    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-sgtxd" event={"ID":"57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4","Type":"ContainerStarted","Data":"87aed90473e60e51cf8453c2f6612573850c7eeab319edaab943898013cbe10d"}
Jul 28 17:17:25.771513 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.771493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j" event={"ID":"8e2c27f5-8271-4f35-b2d1-b34649409743","Type":"ContainerStarted","Data":"c512ec4749d5418981b3540409bcbb7c91be5cb3b635a52391ee79cd71d4e5ef"}
Jul 28 17:17:25.773054 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.773036    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qh26v" event={"ID":"9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c","Type":"ContainerStarted","Data":"1167f42a1d882d47463238ba12dcef573bac38612b428cba20d1e7732fdbb188"}
Jul 28 17:17:25.804649 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.804592    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-sgtxd" podStartSLOduration=3.153296682 podStartE2EDuration="21.804574842s" totalImagesPullingTime="18.65127816s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:25.803862085 +0000 UTC m=+22.737261503" watchObservedRunningTime="2026-07-28 17:17:25.804574842 +0000 UTC m=+22.737974260"
Jul 28 17:17:25.819202 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.819155    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6b6qm" podStartSLOduration=3.185590105 podStartE2EDuration="21.819137437s" totalImagesPullingTime="18.633547332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:25.818913674 +0000 UTC m=+22.752313093" watchObservedRunningTime="2026-07-28 17:17:25.819137437 +0000 UTC m=+22.752536855"
Jul 28 17:17:25.836170 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.836127    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cn5vl" podStartSLOduration=7.252364233 podStartE2EDuration="21.836113572s" totalImagesPullingTime="14.583749339s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:25.836052058 +0000 UTC m=+22.769451476" watchObservedRunningTime="2026-07-28 17:17:25.836113572 +0000 UTC m=+22.769512990"
Jul 28 17:17:25.873128 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:25.873078    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qh26v" podStartSLOduration=3.236355437 podStartE2EDuration="21.873058715s" totalImagesPullingTime="18.636703278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:25.851638713 +0000 UTC m=+22.785038130" watchObservedRunningTime="2026-07-28 17:17:25.873058715 +0000 UTC m=+22.806458134"
Jul 28 17:17:26.036017 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:26.035985    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cn5vl_767e90f1-3aaa-4ec1-adb0-e732020844d5/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:26.100881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:26.100854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cn5vl_767e90f1-3aaa-4ec1-adb0-e732020844d5/konnectivity-agent/0.log" limit=1024
Jul 28 17:17:26.591666 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:26.591633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:26.592616 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:26.591786    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:26.778687 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:26.778640    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"53091bd6dba40939e031808a1aac9536bad3889f26365a605fe32834fc62c948"}
Jul 28 17:17:26.778814 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:26.778744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"8bd44a9b6c428389289e5d9db9c063ce7a0cd6cca09daa0ce26b8e41728603f6"}
Jul 28 17:17:26.778814 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:26.778796    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"d75b5af83eb42627a0a50794d49971cd660a58e78d06c89467224e46e40795d2"}
Jul 28 17:17:26.778899 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:26.778842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"977457fdd7717edf07c5f34a0b0380d716cb703ab9690c41849bf1fbe6d2d1cd"}
Jul 28 17:17:26.915570 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:26.915542    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 17:17:27.466145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.466121    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:27.466740 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.466721    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:27.487684 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.487618    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-wh264" podStartSLOduration=4.7792000980000005 podStartE2EDuration="23.487600213s" totalImagesPullingTime="18.708400115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:25.873843552 +0000 UTC m=+22.807242970" watchObservedRunningTime="2026-07-28 17:17:27.487600213 +0000 UTC m=+24.420999632"
Jul 28 17:17:27.515477 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.515381    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T17:17:26.915564502Z","UUID":"669dd614-987b-4c5b-8726-159dfb0d1141","Handler":null,"Name":"","Endpoint":""}
Jul 28 17:17:27.516970 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.516951    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 17:17:27.517082 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.516992    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 17:17:27.591918 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.591891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:27.592385 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.591891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:27.592385 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:27.592028    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:27.592385 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:27.592083    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:27.783622 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.783543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j" event={"ID":"8e2c27f5-8271-4f35-b2d1-b34649409743","Type":"ContainerStarted","Data":"18405cefd2b15422586553ce2aca80ebffe26dfc0d2414b0be6297789936fcc5"}
Jul 28 17:17:27.785278 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.785247    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-z4k8t" event={"ID":"79946742-16f6-4416-a9d2-e54871e8ed2d","Type":"ContainerStarted","Data":"5de4cd7d38c15a1821b7db2551343fac4f5f013443dfa25bbbfd06ec377283d7"}
Jul 28 17:17:27.785566 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.785545    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:27.785987 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.785968    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cn5vl"
Jul 28 17:17:27.819117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:27.819068    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-z4k8t" podStartSLOduration=5.176501337 podStartE2EDuration="23.819050087s" totalImagesPullingTime="18.64254875s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:27.803147163 +0000 UTC m=+24.736546582" watchObservedRunningTime="2026-07-28 17:17:27.819050087 +0000 UTC m=+24.752449505"
Jul 28 17:17:28.591788 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:28.591716    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:28.591936 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:28.591818    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:28.791407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:28.791371    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"3fbb5c14041e6f7cdf7fabed3e26fabb28c724eda24993859d10815fc60f2a2e"}
Jul 28 17:17:29.591471 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:29.591444    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:29.591624 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:29.591550    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:29.591719 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:29.591617    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:29.591789 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:29.591768    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:30.591345 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.591220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:30.591897 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:30.591422    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:30.797947 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.797712    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" event={"ID":"17c64fb3-f111-4c84-9b47-6c4695055d2f","Type":"ContainerStarted","Data":"b80c4cf7e8d45fa3acc78c5c72ef589421b0970f591b5073943b32ff70b89211"}
Jul 28 17:17:30.797947 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.797927    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:30.799873 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.799841    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j" event={"ID":"8e2c27f5-8271-4f35-b2d1-b34649409743","Type":"ContainerStarted","Data":"8202b4abe7d0a57dcbe9498c3dd68ed5086d362fdf9ce1cf7fcf63aa20b390ea"}
Jul 28 17:17:30.801619 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.801594    2563 generic.go:356] "Generic (PLEG): container finished" podID="a728637b-8a7c-4a0a-bb2d-b9b18c258162" containerID="32168017c3cada9760620f0b89b1dd10c182d8f765da51d099142e1dc04c8d46" exitCode=0
Jul 28 17:17:30.801748 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.801623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" event={"ID":"a728637b-8a7c-4a0a-bb2d-b9b18c258162","Type":"ContainerDied","Data":"32168017c3cada9760620f0b89b1dd10c182d8f765da51d099142e1dc04c8d46"}
Jul 28 17:17:30.813638 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.813620    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:30.867985 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.867946    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp" podStartSLOduration=8.167260515 podStartE2EDuration="26.867935097s" totalImagesPullingTime="18.700674582s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:30.829103594 +0000 UTC m=+27.762502998" watchObservedRunningTime="2026-07-28 17:17:30.867935097 +0000 UTC m=+27.801334515"
Jul 28 17:17:30.888775 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:30.888738    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-h944j" podStartSLOduration=3.9705203620000002 podStartE2EDuration="26.888727598s" totalImagesPullingTime="22.918207236s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:30.888667618 +0000 UTC m=+27.822067036" watchObservedRunningTime="2026-07-28 17:17:30.888727598 +0000 UTC m=+27.822127016"
Jul 28 17:17:31.591447 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:31.591380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:31.591826 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:31.591515    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:31.591826 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:31.591561    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:31.591826 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:31.591642    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:31.804363 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:31.804330    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:31.804527 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:31.804428    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:31.818718 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:31.818694    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:17:32.495642 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:32.495611    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mlbvr"]
Jul 28 17:17:32.495948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:32.495768    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:32.495948 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:32.495847    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:32.497932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:32.497907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rlbl4"]
Jul 28 17:17:32.498061 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:32.498004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:32.498125 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:32.498099    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:32.498756 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:32.498730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjl4f"]
Jul 28 17:17:32.498860 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:32.498842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:32.498966 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:32.498944    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:32.808649 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:32.808613    2563 generic.go:356] "Generic (PLEG): container finished" podID="a728637b-8a7c-4a0a-bb2d-b9b18c258162" containerID="a4ac8b449b2d8b7e938e9f47047b61c6e3c9910c76d28f3f6def3cbfb339f501" exitCode=0
Jul 28 17:17:32.809079 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:32.808704    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" event={"ID":"a728637b-8a7c-4a0a-bb2d-b9b18c258162","Type":"ContainerDied","Data":"a4ac8b449b2d8b7e938e9f47047b61c6e3c9910c76d28f3f6def3cbfb339f501"}
Jul 28 17:17:34.592075 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:34.591904    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:34.592443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:34.591955    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:34.592443 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:34.592154    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:34.592443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:34.591956    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:34.592443 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:34.592234    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:34.592443 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:34.592298    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:34.816008 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:34.815942    2563 generic.go:356] "Generic (PLEG): container finished" podID="a728637b-8a7c-4a0a-bb2d-b9b18c258162" containerID="faae2e0e606070c170a264aba2d2e0b84f8a2c6c0d8c4afffcc59cb87469d521" exitCode=0
Jul 28 17:17:34.816008 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:34.815978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" event={"ID":"a728637b-8a7c-4a0a-bb2d-b9b18c258162","Type":"ContainerDied","Data":"faae2e0e606070c170a264aba2d2e0b84f8a2c6c0d8c4afffcc59cb87469d521"}
Jul 28 17:17:35.714575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:35.714545    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qh26v_9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c/dns-node-resolver/0.log"
Jul 28 17:17:36.315776 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:36.315748    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6b6qm_ccbf21fa-84d8-4348-bc89-36ea89e9284f/node-ca/0.log"
Jul 28 17:17:36.591578 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:36.591548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:36.591758 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:36.591592    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:36.591758 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:36.591592    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:36.591758 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:36.591695    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gjl4f" podUID="42b3b5e9-0f43-4c50-be37-a7c61ff99746"
Jul 28 17:17:36.591900 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:36.591774    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rlbl4" podUID="26a3cd7c-de8f-4208-b600-b598dd82afd4"
Jul 28 17:17:36.591900 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:36.591856    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-mlbvr" podUID="94714598-e34a-4953-b07d-c1b9037a561e"
Jul 28 17:17:37.216918 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:37.216878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:37.217419 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:37.217005    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:37.217419 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:37.217059    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret podName:42b3b5e9-0f43-4c50-be37-a7c61ff99746 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:09.217042024 +0000 UTC m=+66.150441420 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret") pod "global-pull-secret-syncer-gjl4f" (UID: "42b3b5e9-0f43-4c50-be37-a7c61ff99746") : object "kube-system"/"original-pull-secret" not registered
Jul 28 17:17:37.318113 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:37.318080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:37.318113 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:37.318112    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:37.318277 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:37.318231    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 17:17:37.318277 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:37.318248    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 17:17:37.318277 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:37.318250    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:37.318277 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:37.318260    2563 projected.go:196] Error preparing data for projected volume kube-api-access-cfxmm for pod openshift-network-diagnostics/network-check-target-mlbvr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:37.318476 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:37.318310    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs podName:26a3cd7c-de8f-4208-b600-b598dd82afd4 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:09.318292761 +0000 UTC m=+66.251692161 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs") pod "network-metrics-daemon-rlbl4" (UID: "26a3cd7c-de8f-4208-b600-b598dd82afd4") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 17:17:37.318476 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:37.318331    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm podName:94714598-e34a-4953-b07d-c1b9037a561e nodeName:}" failed. No retries permitted until 2026-07-28 17:18:09.318321072 +0000 UTC m=+66.251720471 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-cfxmm" (UniqueName: "kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm") pod "network-check-target-mlbvr" (UID: "94714598-e34a-4953-b07d-c1b9037a561e") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 17:17:38.376235 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.376209    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-137-64.ec2.internal" event="NodeReady"
Jul 28 17:17:38.376792 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.376341    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 17:17:38.430090 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.430044    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77b5b5d789-5mprw"]
Jul 28 17:17:38.432627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.432607    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.435210 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.435187    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 17:17:38.435210 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.435208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-qp66m\""
Jul 28 17:17:38.435346 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.435291    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 17:17:38.436011 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.435989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 17:17:38.443976 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.443957    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 17:17:38.444647 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.444628    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5sv95"]
Jul 28 17:17:38.446497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.446478    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.450761 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.450742    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 17:17:38.452218 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.452196    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qlxg9"]
Jul 28 17:17:38.455994 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.453688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-9lnwf\""
Jul 28 17:17:38.455994 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.453891    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 17:17:38.457220 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.457199    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77b5b5d789-5mprw"]
Jul 28 17:17:38.457375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.457336    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.460364 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.460342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 17:17:38.460473 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.460416    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 17:17:38.460606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.460590    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5lhcj\""
Jul 28 17:17:38.460719 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.460613    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 17:17:38.460719 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.460643    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 17:17:38.463858 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.463449    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5sv95"]
Jul 28 17:17:38.473927 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.473906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qlxg9"]
Jul 28 17:17:38.479484 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.479464    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-zm59x"]
Jul 28 17:17:38.481486 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.481431    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zm59x"
Jul 28 17:17:38.484270 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.484247    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 17:17:38.484489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.484474    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 17:17:38.488113 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.488082    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 17:17:38.488298 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.488285    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-jwr5z\""
Jul 28 17:17:38.498099 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.497954    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zm59x"]
Jul 28 17:17:38.537427 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.537402    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg"]
Jul 28 17:17:38.539701 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.539664    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg"
Jul 28 17:17:38.542509 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.542474    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 28 17:17:38.543519 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.543238    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 28 17:17:38.543519 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.543336    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-cc7km\""
Jul 28 17:17:38.551010 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.550987    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg"]
Jul 28 17:17:38.591427 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.591401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:17:38.591585 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.591533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:17:38.591585 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.591563    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:17:38.594391 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.594370    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:17:38.596108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.596086    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hrst7\""
Jul 28 17:17:38.596529 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.596512    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:17:38.596765 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.596751    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c94ch\""
Jul 28 17:17:38.596841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.596774    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:17:38.596841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.596777    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:17:38.627927 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.627875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-bound-sa-token\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.627927 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.627904    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.628089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.627925    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tt9vk\" (UniqueName: \"kubernetes.io/projected/53101351-b715-4d1c-9072-67c1165ede98-kube-api-access-tt9vk\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.628089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.627960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.628089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.627981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20a8a35-85b3-4c80-98f7-55055e247acb-tmp-dir\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.628089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.627997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-ca-trust-extracted\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.628089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-installation-pull-secrets\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.628089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.628089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628060    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ffhj\" (UniqueName: \"kubernetes.io/projected/c20a8a35-85b3-4c80-98f7-55055e247acb-kube-api-access-5ffhj\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7m6c\" (UniqueName: \"kubernetes.io/projected/272d8332-5d3c-4556-bcc6-57b9a2f9a946-kube-api-access-j7m6c\") pod \"ingress-canary-zm59x\" (UID: \"272d8332-5d3c-4556-bcc6-57b9a2f9a946\") " pod="openshift-ingress-canary/ingress-canary-zm59x"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628158    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53101351-b715-4d1c-9072-67c1165ede98-data-volume\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628229    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c20a8a35-85b3-4c80-98f7-55055e247acb-config-volume\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628245    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-image-registry-private-configuration\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628301    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-certificates\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-trusted-ca\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628356    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/272d8332-5d3c-4556-bcc6-57b9a2f9a946-cert\") pod \"ingress-canary-zm59x\" (UID: \"272d8332-5d3c-4556-bcc6-57b9a2f9a946\") " pod="openshift-ingress-canary/ingress-canary-zm59x"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628390    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53101351-b715-4d1c-9072-67c1165ede98-crio-socket\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.628440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.628409    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2969t\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-kube-api-access-2969t\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.729289 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729259    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-bound-sa-token\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.729289 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.729507 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729322    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tt9vk\" (UniqueName: \"kubernetes.io/projected/53101351-b715-4d1c-9072-67c1165ede98-kube-api-access-tt9vk\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.729507 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2h6ql\" (UniqueName: \"kubernetes.io/projected/ff883792-b637-477e-a327-f0e44129405b-kube-api-access-2h6ql\") pod \"migrator-6865c9967-qc9gg\" (UID: \"ff883792-b637-477e-a327-f0e44129405b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg"
Jul 28 17:17:38.729507 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729379    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.729507 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.729400    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:38.729507 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.729484    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:39.229461777 +0000 UTC m=+36.162861183 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:38.729756 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.729581    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:38.729756 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729404    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20a8a35-85b3-4c80-98f7-55055e247acb-tmp-dir\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.729756 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.729638    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:39.22961926 +0000 UTC m=+36.163018673 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:38.729756 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-ca-trust-extracted\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.729756 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c20a8a35-85b3-4c80-98f7-55055e247acb-tmp-dir\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.729756 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-installation-pull-secrets\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.729756 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729755    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729780    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ffhj\" (UniqueName: \"kubernetes.io/projected/c20a8a35-85b3-4c80-98f7-55055e247acb-kube-api-access-5ffhj\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7m6c\" (UniqueName: \"kubernetes.io/projected/272d8332-5d3c-4556-bcc6-57b9a2f9a946-kube-api-access-j7m6c\") pod \"ingress-canary-zm59x\" (UID: \"272d8332-5d3c-4556-bcc6-57b9a2f9a946\") " pod="openshift-ingress-canary/ingress-canary-zm59x"
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53101351-b715-4d1c-9072-67c1165ede98-data-volume\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.729888    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c20a8a35-85b3-4c80-98f7-55055e247acb-config-volume\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.729943    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls podName:c20a8a35-85b3-4c80-98f7-55055e247acb nodeName:}" failed. No retries permitted until 2026-07-28 17:17:39.22992828 +0000 UTC m=+36.163327691 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls") pod "dns-default-5sv95" (UID: "c20a8a35-85b3-4c80-98f7-55055e247acb") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.729970    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-image-registry-private-configuration\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.730026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730003    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-ca-trust-extracted\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.730375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730032    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-certificates\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.730375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-trusted-ca\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.730375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/272d8332-5d3c-4556-bcc6-57b9a2f9a946-cert\") pod \"ingress-canary-zm59x\" (UID: \"272d8332-5d3c-4556-bcc6-57b9a2f9a946\") " pod="openshift-ingress-canary/ingress-canary-zm59x"
Jul 28 17:17:38.730375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53101351-b715-4d1c-9072-67c1165ede98-crio-socket\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.730375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2969t\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-kube-api-access-2969t\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.730554 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/c20a8a35-85b3-4c80-98f7-55055e247acb-config-volume\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.730554 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730526    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/53101351-b715-4d1c-9072-67c1165ede98-data-volume\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.730554 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.730550    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:38.730651 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.730564    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77b5b5d789-5mprw: secret "image-registry-tls" not found
Jul 28 17:17:38.730651 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:38.730609    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls podName:bdbf6db7-c494-4ec0-983c-5e90bd957f9c nodeName:}" failed. No retries permitted until 2026-07-28 17:17:39.2305934 +0000 UTC m=+36.163992796 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls") pod "image-registry-77b5b5d789-5mprw" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c") : secret "image-registry-tls" not found
Jul 28 17:17:38.731010 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.730961    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/53101351-b715-4d1c-9072-67c1165ede98-crio-socket\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.731165 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.731124    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-certificates\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.731454 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.731432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-trusted-ca\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.734431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.734403    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-installation-pull-secrets\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.734522 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.734410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-image-registry-private-configuration\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.734522 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.734508    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/272d8332-5d3c-4556-bcc6-57b9a2f9a946-cert\") pod \"ingress-canary-zm59x\" (UID: \"272d8332-5d3c-4556-bcc6-57b9a2f9a946\") " pod="openshift-ingress-canary/ingress-canary-zm59x"
Jul 28 17:17:38.740620 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.740597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tt9vk\" (UniqueName: \"kubernetes.io/projected/53101351-b715-4d1c-9072-67c1165ede98-kube-api-access-tt9vk\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:38.741606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.741355    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ffhj\" (UniqueName: \"kubernetes.io/projected/c20a8a35-85b3-4c80-98f7-55055e247acb-kube-api-access-5ffhj\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:38.741606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.741595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-bound-sa-token\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.742188 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.742160    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7m6c\" (UniqueName: \"kubernetes.io/projected/272d8332-5d3c-4556-bcc6-57b9a2f9a946-kube-api-access-j7m6c\") pod \"ingress-canary-zm59x\" (UID: \"272d8332-5d3c-4556-bcc6-57b9a2f9a946\") " pod="openshift-ingress-canary/ingress-canary-zm59x"
Jul 28 17:17:38.742279 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.742170    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2969t\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-kube-api-access-2969t\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:38.791207 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.791186    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-zm59x"
Jul 28 17:17:38.830854 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.830820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2h6ql\" (UniqueName: \"kubernetes.io/projected/ff883792-b637-477e-a327-f0e44129405b-kube-api-access-2h6ql\") pod \"migrator-6865c9967-qc9gg\" (UID: \"ff883792-b637-477e-a327-f0e44129405b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg"
Jul 28 17:17:38.840847 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.840822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2h6ql\" (UniqueName: \"kubernetes.io/projected/ff883792-b637-477e-a327-f0e44129405b-kube-api-access-2h6ql\") pod \"migrator-6865c9967-qc9gg\" (UID: \"ff883792-b637-477e-a327-f0e44129405b\") " pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg"
Jul 28 17:17:38.849665 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:38.849635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg"
Jul 28 17:17:39.183549 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.183433    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-zm59x"]
Jul 28 17:17:39.186423 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:39.186397    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod272d8332_5d3c_4556_bcc6_57b9a2f9a946.slice/crio-fb8be68e15d049068ccf39890d13b75da16b3f11ba429b1d0d09b196e1963d31 WatchSource:0}: Error finding container fb8be68e15d049068ccf39890d13b75da16b3f11ba429b1d0d09b196e1963d31: Status 404 returned error can't find the container with id fb8be68e15d049068ccf39890d13b75da16b3f11ba429b1d0d09b196e1963d31
Jul 28 17:17:39.189280 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.189196    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg"]
Jul 28 17:17:39.191954 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:39.191931    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff883792_b637_477e_a327_f0e44129405b.slice/crio-96be29c50c5bd7a32abf2dd7c5a73f1bb374e910b60ec8e7812595bb4c02dfc1 WatchSource:0}: Error finding container 96be29c50c5bd7a32abf2dd7c5a73f1bb374e910b60ec8e7812595bb4c02dfc1: Status 404 returned error can't find the container with id 96be29c50c5bd7a32abf2dd7c5a73f1bb374e910b60ec8e7812595bb4c02dfc1
Jul 28 17:17:39.234007 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.233975    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:39.234130 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.234017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:39.234130 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234112    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:39.234130 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234113    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:39.234280 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.234152    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:39.234280 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234161    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls podName:c20a8a35-85b3-4c80-98f7-55055e247acb nodeName:}" failed. No retries permitted until 2026-07-28 17:17:40.234146496 +0000 UTC m=+37.167545893 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls") pod "dns-default-5sv95" (UID: "c20a8a35-85b3-4c80-98f7-55055e247acb") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:39.234280 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234225    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:40.234204693 +0000 UTC m=+37.167604092 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:39.234280 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234250    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:39.234280 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234264    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77b5b5d789-5mprw: secret "image-registry-tls" not found
Jul 28 17:17:39.234522 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234313    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls podName:bdbf6db7-c494-4ec0-983c-5e90bd957f9c nodeName:}" failed. No retries permitted until 2026-07-28 17:17:40.234298872 +0000 UTC m=+37.167698269 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls") pod "image-registry-77b5b5d789-5mprw" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c") : secret "image-registry-tls" not found
Jul 28 17:17:39.234522 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.234351    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:39.234522 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234427    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:39.234522 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:39.234481    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:40.23446912 +0000 UTC m=+37.167868517 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:39.831721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.831650    2563 generic.go:356] "Generic (PLEG): container finished" podID="a728637b-8a7c-4a0a-bb2d-b9b18c258162" containerID="4b2c2c2fed5106d768e82b200b0cf40f395e28d386afd0dafb46553fe0fdf3e1" exitCode=0
Jul 28 17:17:39.831721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.831711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" event={"ID":"a728637b-8a7c-4a0a-bb2d-b9b18c258162","Type":"ContainerDied","Data":"4b2c2c2fed5106d768e82b200b0cf40f395e28d386afd0dafb46553fe0fdf3e1"}
Jul 28 17:17:39.833463 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.833440    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zm59x" event={"ID":"272d8332-5d3c-4556-bcc6-57b9a2f9a946","Type":"ContainerStarted","Data":"fb8be68e15d049068ccf39890d13b75da16b3f11ba429b1d0d09b196e1963d31"}
Jul 28 17:17:39.834640 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:39.834614    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg" event={"ID":"ff883792-b637-477e-a327-f0e44129405b","Type":"ContainerStarted","Data":"96be29c50c5bd7a32abf2dd7c5a73f1bb374e910b60ec8e7812595bb4c02dfc1"}
Jul 28 17:17:40.241533 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.241451    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:40.241704 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.241557    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:40.241704 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.241593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:40.241704 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.241631    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:42.241611222 +0000 UTC m=+39.175010630 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:40.241860 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.241711    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:40.241860 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.241756    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:42.241742066 +0000 UTC m=+39.175141462 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:40.241860 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.241774    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:40.241860 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.241800    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:40.241986 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.241911    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:40.241986 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.241926    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77b5b5d789-5mprw: secret "image-registry-tls" not found
Jul 28 17:17:40.241986 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.241955    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls podName:bdbf6db7-c494-4ec0-983c-5e90bd957f9c nodeName:}" failed. No retries permitted until 2026-07-28 17:17:42.241946938 +0000 UTC m=+39.175346335 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls") pod "image-registry-77b5b5d789-5mprw" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c") : secret "image-registry-tls" not found
Jul 28 17:17:40.242112 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.241995    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:40.242112 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:40.242024    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls podName:c20a8a35-85b3-4c80-98f7-55055e247acb nodeName:}" failed. No retries permitted until 2026-07-28 17:17:42.242014168 +0000 UTC m=+39.175413580 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls") pod "dns-default-5sv95" (UID: "c20a8a35-85b3-4c80-98f7-55055e247acb") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:40.839278 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.839244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg" event={"ID":"ff883792-b637-477e-a327-f0e44129405b","Type":"ContainerStarted","Data":"6694a42fe1ae2a13e4d25bd732d5e1ae6b4128889e50dc0118bd3c5b397c33e5"}
Jul 28 17:17:40.839721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.839300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg" event={"ID":"ff883792-b637-477e-a327-f0e44129405b","Type":"ContainerStarted","Data":"abe267956458c8331a00bac41880ffb406f03a851be12018dffb303b42258273"}
Jul 28 17:17:40.842660 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.842636    2563 generic.go:356] "Generic (PLEG): container finished" podID="a728637b-8a7c-4a0a-bb2d-b9b18c258162" containerID="0eb0d2b667e28a19fa6b82be13d08731182171c1946be30267003bc5196bd1e8" exitCode=0
Jul 28 17:17:40.842794 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.842690    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" event={"ID":"a728637b-8a7c-4a0a-bb2d-b9b18c258162","Type":"ContainerDied","Data":"0eb0d2b667e28a19fa6b82be13d08731182171c1946be30267003bc5196bd1e8"}
Jul 28 17:17:40.856777 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:40.856720    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-6865c9967-qc9gg" podStartSLOduration=1.7395635280000001 podStartE2EDuration="2.856702914s" totalImagesPullingTime="1.117139386s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:40.856543322 +0000 UTC m=+37.789942746" watchObservedRunningTime="2026-07-28 17:17:40.856702914 +0000 UTC m=+37.790102335"
Jul 28 17:17:41.848499 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:41.848303    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" event={"ID":"a728637b-8a7c-4a0a-bb2d-b9b18c258162","Type":"ContainerStarted","Data":"776370026ff96366003d2851e6e9c90ce741c5066e88b70b0e5696c36eac05c0"}
Jul 28 17:17:41.849837 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:41.849815    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-zm59x" event={"ID":"272d8332-5d3c-4556-bcc6-57b9a2f9a946","Type":"ContainerStarted","Data":"772770f5d8567be616a6636c66e32627af8faedea1a32925f9232ef8f85b8daf"}
Jul 28 17:17:41.877418 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:41.877376    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-dtzv7" podStartSLOduration=8.24862246 podStartE2EDuration="37.877362608s" totalImagesPullingTime="29.628740148s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:41.876886497 +0000 UTC m=+38.810285920" watchObservedRunningTime="2026-07-28 17:17:41.877362608 +0000 UTC m=+38.810762026"
Jul 28 17:17:41.899144 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:41.899103    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-zm59x" podStartSLOduration=1.5124567519999998 podStartE2EDuration="3.899089622s" totalImagesPullingTime="2.38663287s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:41.898522829 +0000 UTC m=+38.831922247" watchObservedRunningTime="2026-07-28 17:17:41.899089622 +0000 UTC m=+38.832489039"
Jul 28 17:17:42.257637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:42.257546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:42.257637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:42.257598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:42.257843 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257716    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:42.257843 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257719    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:42.257843 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257788    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77b5b5d789-5mprw: secret "image-registry-tls" not found
Jul 28 17:17:42.257843 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257794    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls podName:c20a8a35-85b3-4c80-98f7-55055e247acb nodeName:}" failed. No retries permitted until 2026-07-28 17:17:46.257772495 +0000 UTC m=+43.191171908 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls") pod "dns-default-5sv95" (UID: "c20a8a35-85b3-4c80-98f7-55055e247acb") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:42.257843 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:42.257832    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:42.257843 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257839    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls podName:bdbf6db7-c494-4ec0-983c-5e90bd957f9c nodeName:}" failed. No retries permitted until 2026-07-28 17:17:46.257828137 +0000 UTC m=+43.191227539 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls") pod "image-registry-77b5b5d789-5mprw" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c") : secret "image-registry-tls" not found
Jul 28 17:17:42.258123 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257884    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:42.258123 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:46.257908368 +0000 UTC m=+43.191307766 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:42.258123 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:42.257883    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:42.258123 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257941    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:42.258123 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:42.257971    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:46.257962448 +0000 UTC m=+43.191361844 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:46.288647 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:46.288610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:46.288652    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:46.288745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288775    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:46.288782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288844    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls podName:c20a8a35-85b3-4c80-98f7-55055e247acb nodeName:}" failed. No retries permitted until 2026-07-28 17:17:54.28882519 +0000 UTC m=+51.222224594 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls") pod "dns-default-5sv95" (UID: "c20a8a35-85b3-4c80-98f7-55055e247acb") : secret "dns-default-metrics-tls" not found
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288847    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288867    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77b5b5d789-5mprw: secret "image-registry-tls" not found
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288868    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288914    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:54.288903865 +0000 UTC m=+51.222303266 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288911    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288928    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls podName:bdbf6db7-c494-4ec0-983c-5e90bd957f9c nodeName:}" failed. No retries permitted until 2026-07-28 17:17:54.288920296 +0000 UTC m=+51.222319694 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls") pod "image-registry-77b5b5d789-5mprw" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c") : secret "image-registry-tls" not found
Jul 28 17:17:46.289122 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:46.288965    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:17:54.288951522 +0000 UTC m=+51.222350917 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : configmap "kube-rbac-proxy" not found
Jul 28 17:17:54.347257 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.347220    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:54.347257 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.347258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:54.347770 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.347313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:54.347770 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.347335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:54.347770 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:54.347499    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:54.347770 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:17:54.347574    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls podName:53101351-b715-4d1c-9072-67c1165ede98 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:10.347554888 +0000 UTC m=+67.280954289 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qlxg9" (UID: "53101351-b715-4d1c-9072-67c1165ede98") : secret "insights-runtime-extractor-tls" not found
Jul 28 17:17:54.347901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.347799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/53101351-b715-4d1c-9072-67c1165ede98-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:17:54.351071 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.351053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/c20a8a35-85b3-4c80-98f7-55055e247acb-metrics-tls\") pod \"dns-default-5sv95\" (UID: \"c20a8a35-85b3-4c80-98f7-55055e247acb\") " pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:54.351149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.351053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"image-registry-77b5b5d789-5mprw\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") " pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:54.357232 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.357204    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:54.509726 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.509696    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5sv95"]
Jul 28 17:17:54.512315 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:54.512287    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc20a8a35_85b3_4c80_98f7_55055e247acb.slice/crio-37b5864a010ed3b86ac0d38f7753df7eb5cc85fdb4451676243bcc14c1922eb2 WatchSource:0}: Error finding container 37b5864a010ed3b86ac0d38f7753df7eb5cc85fdb4451676243bcc14c1922eb2: Status 404 returned error can't find the container with id 37b5864a010ed3b86ac0d38f7753df7eb5cc85fdb4451676243bcc14c1922eb2
Jul 28 17:17:54.643531 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.643467    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:54.781805 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.781716    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77b5b5d789-5mprw"]
Jul 28 17:17:54.784035 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:17:54.784007    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdbf6db7_c494_4ec0_983c_5e90bd957f9c.slice/crio-6d9b555a145635611dfc500525ee1f71285ff80ac7b7fcf2f935dd6f62e946a3 WatchSource:0}: Error finding container 6d9b555a145635611dfc500525ee1f71285ff80ac7b7fcf2f935dd6f62e946a3: Status 404 returned error can't find the container with id 6d9b555a145635611dfc500525ee1f71285ff80ac7b7fcf2f935dd6f62e946a3
Jul 28 17:17:54.881066 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.881041    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw" event={"ID":"bdbf6db7-c494-4ec0-983c-5e90bd957f9c","Type":"ContainerStarted","Data":"6d9b555a145635611dfc500525ee1f71285ff80ac7b7fcf2f935dd6f62e946a3"}
Jul 28 17:17:54.881973 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:54.881946    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5sv95" event={"ID":"c20a8a35-85b3-4c80-98f7-55055e247acb","Type":"ContainerStarted","Data":"37b5864a010ed3b86ac0d38f7753df7eb5cc85fdb4451676243bcc14c1922eb2"}
Jul 28 17:17:55.888554 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:55.888519    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw" event={"ID":"bdbf6db7-c494-4ec0-983c-5e90bd957f9c","Type":"ContainerStarted","Data":"65eb26f5da5174845a475616f3ab4f3faa66b34861194fe47da99c2c49e6b8f7"}
Jul 28 17:17:55.888926 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:55.888609    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:17:55.918538 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:55.918493    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw" podStartSLOduration=31.91848015 podStartE2EDuration="31.91848015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:55.91693347 +0000 UTC m=+52.850332890" watchObservedRunningTime="2026-07-28 17:17:55.91848015 +0000 UTC m=+52.851879567"
Jul 28 17:17:57.895975 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:57.895939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5sv95" event={"ID":"c20a8a35-85b3-4c80-98f7-55055e247acb","Type":"ContainerStarted","Data":"e8c664ab336b084663c70425f30938b4534a98b99b6c458b4ea12259e2f5f4bc"}
Jul 28 17:17:57.896349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:57.895986    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5sv95" event={"ID":"c20a8a35-85b3-4c80-98f7-55055e247acb","Type":"ContainerStarted","Data":"ed9c6fe96588e7a9a8bc3371e5d119217b94a813de46f0bcfa25d847a88a726c"}
Jul 28 17:17:57.896349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:57.896067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5sv95"
Jul 28 17:17:57.920588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:17:57.920549    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5sv95" podStartSLOduration=17.631624802 podStartE2EDuration="19.920536803s" totalImagesPullingTime="2.288912001s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:17:57.919898093 +0000 UTC m=+54.853297512" watchObservedRunningTime="2026-07-28 17:17:57.920536803 +0000 UTC m=+54.853936221"
Jul 28 17:18:03.541100 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:03.541070    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77b5b5d789-5mprw"]
Jul 28 17:18:03.824795 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:03.824776    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-56rvp"
Jul 28 17:18:07.003081 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.003050    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2484n"]
Jul 28 17:18:07.005895 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.005880    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.009866 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.009839    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 17:18:07.009990 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.009952    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 28 17:18:07.009990 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.009971    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-g4lj6\""
Jul 28 17:18:07.009990 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.009970    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 28 17:18:07.010163 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.009995    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 17:18:07.010163 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.010062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 17:18:07.022388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.022367    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2484n"]
Jul 28 17:18:07.143060 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.143023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fltxd\" (UniqueName: \"kubernetes.io/projected/b2abc20c-0068-4bbc-886f-75228cca03ac-kube-api-access-fltxd\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.143218 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.143083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.143218 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.143125    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.143218 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.143165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2abc20c-0068-4bbc-886f-75228cca03ac-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.244306 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.244275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.244391 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.244313    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.244391 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.244333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2abc20c-0068-4bbc-886f-75228cca03ac-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.244391 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.244381    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fltxd\" (UniqueName: \"kubernetes.io/projected/b2abc20c-0068-4bbc-886f-75228cca03ac-kube-api-access-fltxd\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.244540 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:18:07.244401    2563 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 28 17:18:07.244540 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:18:07.244450    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-tls podName:b2abc20c-0068-4bbc-886f-75228cca03ac nodeName:}" failed. No retries permitted until 2026-07-28 17:18:07.744435517 +0000 UTC m=+64.677834916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-tls") pod "prometheus-operator-84d69f794d-2484n" (UID: "b2abc20c-0068-4bbc-886f-75228cca03ac") : secret "prometheus-operator-tls" not found
Jul 28 17:18:07.245017 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.244995    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b2abc20c-0068-4bbc-886f-75228cca03ac-metrics-client-ca\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.246466 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.246449    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.253008 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.252991    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fltxd\" (UniqueName: \"kubernetes.io/projected/b2abc20c-0068-4bbc-886f-75228cca03ac-kube-api-access-fltxd\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.748521 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.748487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.750726 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.750701    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/b2abc20c-0068-4bbc-886f-75228cca03ac-prometheus-operator-tls\") pod \"prometheus-operator-84d69f794d-2484n\" (UID: \"b2abc20c-0068-4bbc-886f-75228cca03ac\") " pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:07.901887 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.901863    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5sv95"
Jul 28 17:18:07.914664 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:07.914647    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n"
Jul 28 17:18:08.037717 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:08.037634    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-84d69f794d-2484n"]
Jul 28 17:18:08.040590 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:08.040566    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2abc20c_0068_4bbc_886f_75228cca03ac.slice/crio-347d2a1e788bc829dcc5126044ad1518913d344eefec6d299e4bfefd9440bf73 WatchSource:0}: Error finding container 347d2a1e788bc829dcc5126044ad1518913d344eefec6d299e4bfefd9440bf73: Status 404 returned error can't find the container with id 347d2a1e788bc829dcc5126044ad1518913d344eefec6d299e4bfefd9440bf73
Jul 28 17:18:08.932958 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:08.932917    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n" event={"ID":"b2abc20c-0068-4bbc-886f-75228cca03ac","Type":"ContainerStarted","Data":"347d2a1e788bc829dcc5126044ad1518913d344eefec6d299e4bfefd9440bf73"}
Jul 28 17:18:09.260217 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.260142    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:18:09.263374 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.263354    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 17:18:09.273133 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.273108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/42b3b5e9-0f43-4c50-be37-a7c61ff99746-original-pull-secret\") pod \"global-pull-secret-syncer-gjl4f\" (UID: \"42b3b5e9-0f43-4c50-be37-a7c61ff99746\") " pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:18:09.361484 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.361446    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:18:09.361641 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.361546    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:18:09.364411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.364385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 17:18:09.364519 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.364388    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 17:18:09.374318 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.374291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/26a3cd7c-de8f-4208-b600-b598dd82afd4-metrics-certs\") pod \"network-metrics-daemon-rlbl4\" (UID: \"26a3cd7c-de8f-4208-b600-b598dd82afd4\") " pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:18:09.374488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.374469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 17:18:09.385176 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.385152    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cfxmm\" (UniqueName: \"kubernetes.io/projected/94714598-e34a-4953-b07d-c1b9037a561e-kube-api-access-cfxmm\") pod \"network-check-target-mlbvr\" (UID: \"94714598-e34a-4953-b07d-c1b9037a561e\") " pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:18:09.503464 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.503435    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gjl4f"
Jul 28 17:18:09.512431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.512408    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-c94ch\""
Jul 28 17:18:09.516525 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.516502    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-hrst7\""
Jul 28 17:18:09.519834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.519811    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:18:09.524582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.524565    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rlbl4"
Jul 28 17:18:09.650310 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.650285    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gjl4f"]
Jul 28 17:18:09.663835 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:09.663808    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42b3b5e9_0f43_4c50_be37_a7c61ff99746.slice/crio-c1c4f3796335a7f2323101dc1c3e0479cbbb08cf7f2f6e49c7619e9579b3e999 WatchSource:0}: Error finding container c1c4f3796335a7f2323101dc1c3e0479cbbb08cf7f2f6e49c7619e9579b3e999: Status 404 returned error can't find the container with id c1c4f3796335a7f2323101dc1c3e0479cbbb08cf7f2f6e49c7619e9579b3e999
Jul 28 17:18:09.676429 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.676401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-mlbvr"]
Jul 28 17:18:09.679454 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:09.679429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94714598_e34a_4953_b07d_c1b9037a561e.slice/crio-a83d3de87d4233ed058699a8bc11368c86e65471ba97a309768d7eac503dd292 WatchSource:0}: Error finding container a83d3de87d4233ed058699a8bc11368c86e65471ba97a309768d7eac503dd292: Status 404 returned error can't find the container with id a83d3de87d4233ed058699a8bc11368c86e65471ba97a309768d7eac503dd292
Jul 28 17:18:09.690453 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.690424    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rlbl4"]
Jul 28 17:18:09.693230 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:09.693147    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26a3cd7c_de8f_4208_b600_b598dd82afd4.slice/crio-0cb564ead5cae6310a81e900b6edc91b74cb55fe6f6f303f1c3cb09965c7c699 WatchSource:0}: Error finding container 0cb564ead5cae6310a81e900b6edc91b74cb55fe6f6f303f1c3cb09965c7c699: Status 404 returned error can't find the container with id 0cb564ead5cae6310a81e900b6edc91b74cb55fe6f6f303f1c3cb09965c7c699
Jul 28 17:18:09.938745 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.938635    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjl4f" event={"ID":"42b3b5e9-0f43-4c50-be37-a7c61ff99746","Type":"ContainerStarted","Data":"c1c4f3796335a7f2323101dc1c3e0479cbbb08cf7f2f6e49c7619e9579b3e999"}
Jul 28 17:18:09.940161 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.940123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlbl4" event={"ID":"26a3cd7c-de8f-4208-b600-b598dd82afd4","Type":"ContainerStarted","Data":"0cb564ead5cae6310a81e900b6edc91b74cb55fe6f6f303f1c3cb09965c7c699"}
Jul 28 17:18:09.941597 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.941544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mlbvr" event={"ID":"94714598-e34a-4953-b07d-c1b9037a561e","Type":"ContainerStarted","Data":"a83d3de87d4233ed058699a8bc11368c86e65471ba97a309768d7eac503dd292"}
Jul 28 17:18:09.943752 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.943731    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n" event={"ID":"b2abc20c-0068-4bbc-886f-75228cca03ac","Type":"ContainerStarted","Data":"ffb1270614b9b33d2ed3564b0b5b03d468bcbbf4560b977fc7d0f4723609bd1e"}
Jul 28 17:18:09.943852 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.943773    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n" event={"ID":"b2abc20c-0068-4bbc-886f-75228cca03ac","Type":"ContainerStarted","Data":"8e420deaa0c12125ebd5b95dc18ea2800c4cd32e0fa0d35a7ab81c4017af2760"}
Jul 28 17:18:09.963321 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:09.963280    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-84d69f794d-2484n" podStartSLOduration=2.383007774 podStartE2EDuration="3.963268595s" totalImagesPullingTime="1.580260821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:09.963029128 +0000 UTC m=+66.896428547" watchObservedRunningTime="2026-07-28 17:18:09.963268595 +0000 UTC m=+66.896668012"
Jul 28 17:18:10.368768 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:10.368310    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:18:10.371310 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:10.371255    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/53101351-b715-4d1c-9072-67c1165ede98-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qlxg9\" (UID: \"53101351-b715-4d1c-9072-67c1165ede98\") " pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:18:10.570758 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:10.570724    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5lhcj\""
Jul 28 17:18:10.578363 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:10.578338    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qlxg9"
Jul 28 17:18:11.033854 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.033824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qlxg9"]
Jul 28 17:18:11.113248 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.113148    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-7c97dc4555-sp2kw"]
Jul 28 17:18:11.130431 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.130270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-sp2kw"
Jul 28 17:18:11.133457 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.133236    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-pr9n7\""
Jul 28 17:18:11.133457 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.133270    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 28 17:18:11.133457 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.133239    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 28 17:18:11.141418 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.141396    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-sp2kw"]
Jul 28 17:18:11.194439 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:11.194408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53101351_b715_4d1c_9072_67c1165ede98.slice/crio-d55550a1bfe43277a9b35c8dc5bf5f123aec0744a72e46f89b456e0b8165bace WatchSource:0}: Error finding container d55550a1bfe43277a9b35c8dc5bf5f123aec0744a72e46f89b456e0b8165bace: Status 404 returned error can't find the container with id d55550a1bfe43277a9b35c8dc5bf5f123aec0744a72e46f89b456e0b8165bace
Jul 28 17:18:11.276017 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.275944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swdgz\" (UniqueName: \"kubernetes.io/projected/da8735fd-e062-458c-933f-7d344a34f479-kube-api-access-swdgz\") pod \"downloads-7c97dc4555-sp2kw\" (UID: \"da8735fd-e062-458c-933f-7d344a34f479\") " pod="openshift-console/downloads-7c97dc4555-sp2kw"
Jul 28 17:18:11.376512 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.376485    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swdgz\" (UniqueName: \"kubernetes.io/projected/da8735fd-e062-458c-933f-7d344a34f479-kube-api-access-swdgz\") pod \"downloads-7c97dc4555-sp2kw\" (UID: \"da8735fd-e062-458c-933f-7d344a34f479\") " pod="openshift-console/downloads-7c97dc4555-sp2kw"
Jul 28 17:18:11.388486 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.388454    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swdgz\" (UniqueName: \"kubernetes.io/projected/da8735fd-e062-458c-933f-7d344a34f479-kube-api-access-swdgz\") pod \"downloads-7c97dc4555-sp2kw\" (UID: \"da8735fd-e062-458c-933f-7d344a34f479\") " pod="openshift-console/downloads-7c97dc4555-sp2kw"
Jul 28 17:18:11.445117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.444999    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-7c97dc4555-sp2kw"
Jul 28 17:18:11.619111 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.619073    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-7c97dc4555-sp2kw"]
Jul 28 17:18:11.623718 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:11.623688    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8735fd_e062_458c_933f_7d344a34f479.slice/crio-5d5936bcbad26c3f0c7235c281215402cf6317985f380923bf0c037d7344fd8c WatchSource:0}: Error finding container 5d5936bcbad26c3f0c7235c281215402cf6317985f380923bf0c037d7344fd8c: Status 404 returned error can't find the container with id 5d5936bcbad26c3f0c7235c281215402cf6317985f380923bf0c037d7344fd8c
Jul 28 17:18:11.954051 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.953985    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-sp2kw" event={"ID":"da8735fd-e062-458c-933f-7d344a34f479","Type":"ContainerStarted","Data":"5d5936bcbad26c3f0c7235c281215402cf6317985f380923bf0c037d7344fd8c"}
Jul 28 17:18:11.956334 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.956304    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlbl4" event={"ID":"26a3cd7c-de8f-4208-b600-b598dd82afd4","Type":"ContainerStarted","Data":"125e5e8dc951d7ebcfcaf658ea6570eeab05f3a48a4d419cda7b07a9f64a3593"}
Jul 28 17:18:11.956464 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.956358    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rlbl4" event={"ID":"26a3cd7c-de8f-4208-b600-b598dd82afd4","Type":"ContainerStarted","Data":"4dafc7d80c721b1b95b750a92bbdd6390dc2faba4cf277a44d4bb4a5569d7dc5"}
Jul 28 17:18:11.957632 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:11.957594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlxg9" event={"ID":"53101351-b715-4d1c-9072-67c1165ede98","Type":"ContainerStarted","Data":"d55550a1bfe43277a9b35c8dc5bf5f123aec0744a72e46f89b456e0b8165bace"}
Jul 28 17:18:12.422575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.421193    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rlbl4" podStartSLOduration=66.893301452 podStartE2EDuration="1m8.42116931s" totalImagesPullingTime="1.527867858s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:11.977706223 +0000 UTC m=+68.911105642" watchObservedRunningTime="2026-07-28 17:18:12.42116931 +0000 UTC m=+69.354568730"
Jul 28 17:18:12.422575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.421370    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-fgpvf"]
Jul 28 17:18:12.436318 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.436151    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.441487 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.441008    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 17:18:12.442937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.441654    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-mkft5\""
Jul 28 17:18:12.442937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.441913    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 17:18:12.445086 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.441746    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 17:18:12.487105 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.487248 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487122    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-tls\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.487248 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-root\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.487248 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487201    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08dc8e97-c2e1-4238-b172-a40801fca1a8-metrics-client-ca\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.487248 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487237    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-textfile\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.487447 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-sys\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.487447 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnsvg\" (UniqueName: \"kubernetes.io/projected/08dc8e97-c2e1-4238-b172-a40801fca1a8-kube-api-access-vnsvg\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.487447 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-wtmp\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.487447 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.487341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.588513 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588487    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-textfile\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.588707 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-sys\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.588707 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588547    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnsvg\" (UniqueName: \"kubernetes.io/projected/08dc8e97-c2e1-4238-b172-a40801fca1a8-kube-api-access-vnsvg\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.588707 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-wtmp\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.588707 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.588707 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.588707 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588657    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-tls\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.589020 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588733    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-root\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.589020 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.588761    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08dc8e97-c2e1-4238-b172-a40801fca1a8-metrics-client-ca\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.589856 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.589244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-wtmp\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.589856 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.589336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/08dc8e97-c2e1-4238-b172-a40801fca1a8-metrics-client-ca\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.589856 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.589475    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-textfile\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.589856 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.589527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-sys\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.589856 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:18:12.589771    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 17:18:12.589856 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.589787    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/08dc8e97-c2e1-4238-b172-a40801fca1a8-root\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.589856 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:18:12.589828    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-tls podName:08dc8e97-c2e1-4238-b172-a40801fca1a8 nodeName:}" failed. No retries permitted until 2026-07-28 17:18:13.089809157 +0000 UTC m=+70.023208566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-tls") pod "node-exporter-fgpvf" (UID: "08dc8e97-c2e1-4238-b172-a40801fca1a8") : secret "node-exporter-tls" not found
Jul 28 17:18:12.590293 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.590188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-accelerators-collector-config\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.592312 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.592259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:12.602789 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:12.602759    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnsvg\" (UniqueName: \"kubernetes.io/projected/08dc8e97-c2e1-4238-b172-a40801fca1a8-kube-api-access-vnsvg\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:13.092934 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.092481    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-tls\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:13.095364 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.095334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/08dc8e97-c2e1-4238-b172-a40801fca1a8-node-exporter-tls\") pod \"node-exporter-fgpvf\" (UID: \"08dc8e97-c2e1-4238-b172-a40801fca1a8\") " pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:13.353437 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.353344    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-fgpvf"
Jul 28 17:18:13.549616 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.549418    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:18:13.744865 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.744770    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:18:13.747755 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.747732    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.751842 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.751760    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:18:13.752881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.752009    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:18:13.752881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.752024    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:18:13.752881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.752268    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ndcpq\""
Jul 28 17:18:13.752881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.752313    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:18:13.752881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.752572    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:18:13.752881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.752796    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:18:13.753331 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.753311    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:18:13.753544 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.753526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:18:13.753660 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.753601    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:18:13.779950 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.779926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:18:13.899850 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.899817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-out\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.899858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ttdh7\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-kube-api-access-ttdh7\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.899888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.899921    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.899967    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-volume\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.899989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-web-config\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900296 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.900050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900643 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.900347    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900643 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.900383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900643 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.900419    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900643 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.900458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900643 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.900497    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:13.900643 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:13.900523    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001453 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001373    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001453 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001429    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001636 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001636 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001548    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001636 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001578    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-out\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001636 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ttdh7\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-kube-api-access-ttdh7\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001653    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-volume\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001782    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-web-config\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.001841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.002041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001834    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.002041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.001876    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.005458 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.005356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.005801 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.005778    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.010366 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.010145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.017606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.011607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.017606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.012923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.017606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.016043    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.021764 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.021291    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-tls-assets\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.022693 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.022630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ttdh7\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-kube-api-access-ttdh7\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.024551 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.024480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-out\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.025016 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.024993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-volume\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.025135 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.024998    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-web-config\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.027277 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.027253    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.027470 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.027435    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:14.061877 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:14.061849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:18:16.771062 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.771024    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"]
Jul 28 17:18:16.795787 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.795755    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"]
Jul 28 17:18:16.795934 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.795881    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:16.800510 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.800456    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-2qjllf4pjm4p4\""
Jul 28 17:18:16.800627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.800459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 28 17:18:16.800627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.800582    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-6fltp\""
Jul 28 17:18:16.800830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.800760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 28 17:18:16.800830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.800666    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 28 17:18:16.800937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.800846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 28 17:18:16.928836 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.928691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b24d701-1478-47b1-b8da-839a8fe0034a-audit-log\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:16.928836 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.928762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-secret-metrics-server-tls\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:16.928836 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.928794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b24d701-1478-47b1-b8da-839a8fe0034a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:16.928836 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.928838    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6h5hq\" (UniqueName: \"kubernetes.io/projected/1b24d701-1478-47b1-b8da-839a8fe0034a-kube-api-access-6h5hq\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:16.929092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.928906    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-secret-metrics-server-client-certs\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:16.929092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.928956    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b24d701-1478-47b1-b8da-839a8fe0034a-metrics-server-audit-profiles\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:16.929092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:16.928980    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-client-ca-bundle\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.029827 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.029753    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6h5hq\" (UniqueName: \"kubernetes.io/projected/1b24d701-1478-47b1-b8da-839a8fe0034a-kube-api-access-6h5hq\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.029827 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.029804    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-secret-metrics-server-client-certs\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.030031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.029846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b24d701-1478-47b1-b8da-839a8fe0034a-metrics-server-audit-profiles\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.030031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.029874    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-client-ca-bundle\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.030031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.029923    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b24d701-1478-47b1-b8da-839a8fe0034a-audit-log\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.030031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.029969    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-secret-metrics-server-tls\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.030031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.029997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b24d701-1478-47b1-b8da-839a8fe0034a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.030811 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.030763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1b24d701-1478-47b1-b8da-839a8fe0034a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.030949 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.030920    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1b24d701-1478-47b1-b8da-839a8fe0034a-audit-log\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.031326 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.031297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1b24d701-1478-47b1-b8da-839a8fe0034a-metrics-server-audit-profiles\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.034234 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.034209    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-client-ca-bundle\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.034502 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.034439    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-secret-metrics-server-tls\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.034972 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.034926    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1b24d701-1478-47b1-b8da-839a8fe0034a-secret-metrics-server-client-certs\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.039106 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.039061    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6h5hq\" (UniqueName: \"kubernetes.io/projected/1b24d701-1478-47b1-b8da-839a8fe0034a-kube-api-access-6h5hq\") pod \"metrics-server-77fcdc95f7-dg5d4\" (UID: \"1b24d701-1478-47b1-b8da-839a8fe0034a\") " pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:17.108814 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:17.108763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:18.145324 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.145232    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:18:18.409578 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:18.409536    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcffed101_7b2e_4bd3_adc0_17bb1f0bb188.slice/crio-00653433931d1fd25099320a085b13556301876f1b12346f2bc178d3ba6242a4 WatchSource:0}: Error finding container 00653433931d1fd25099320a085b13556301876f1b12346f2bc178d3ba6242a4: Status 404 returned error can't find the container with id 00653433931d1fd25099320a085b13556301876f1b12346f2bc178d3ba6242a4
Jul 28 17:18:18.412045 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:18.412018    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08dc8e97_c2e1_4238_b172_a40801fca1a8.slice/crio-8d8413f7543c2ed16b2581bd0dbab42a7e77a6cbcbeacee87d3d56518ecd06b8 WatchSource:0}: Error finding container 8d8413f7543c2ed16b2581bd0dbab42a7e77a6cbcbeacee87d3d56518ecd06b8: Status 404 returned error can't find the container with id 8d8413f7543c2ed16b2581bd0dbab42a7e77a6cbcbeacee87d3d56518ecd06b8
Jul 28 17:18:18.576778 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.576744    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"]
Jul 28 17:18:18.585231 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:18.585187    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b24d701_1478_47b1_b8da_839a8fe0034a.slice/crio-a144b0d0e78d09518bdbc2c4523ab75217a30f4842ac07fa0433660c484514fb WatchSource:0}: Error finding container a144b0d0e78d09518bdbc2c4523ab75217a30f4842ac07fa0433660c484514fb: Status 404 returned error can't find the container with id a144b0d0e78d09518bdbc2c4523ab75217a30f4842ac07fa0433660c484514fb
Jul 28 17:18:18.757630 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.757550    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:18.761425 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.760398    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.764514 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.764485    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:18:18.764646 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.764536    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-euqfncq37bf1t\""
Jul 28 17:18:18.764887 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.764802    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:18:18.765643 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.765612    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:18:18.765872 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.765805    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.766128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.766183    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.766262    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.766298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.766475    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.766506    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.766547    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.766728    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:18:18.769734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.768121    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-48p8q\""
Jul 28 17:18:18.770531 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.770364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:18:18.772855 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.772580    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:18:18.792731 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.792706    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:18.853402 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853564 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853415    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853564 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853446    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853564 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853564 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853564 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853564 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853562    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853615    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5tzf\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-kube-api-access-f5tzf\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853716    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853769    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853792    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-config\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853852    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.853889 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853875    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.854356 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-config-out\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.854356 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.854356 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.854356 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.853976    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-web-config\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-config\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955337    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-config-out\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955457    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-web-config\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955623    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955649    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5tzf\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-kube-api-access-f5tzf\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.956721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.957657 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955858    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.957657 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955881    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.957657 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.955906    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.961319 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.957806    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.961319 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.958919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.961319 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.960571    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.961319 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.961094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.962077 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.962053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.962360 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.962338    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.966973 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.966612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.967531 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.967502    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.969706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.968473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.971745 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.971226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.971892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.971853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.979261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.977827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-config-out\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.979261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.978056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.979261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.978229    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-config\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.979261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.978500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.979261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.978648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.979261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.978909    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-web-config\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.979261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.979222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.980657 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.979799    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5tzf\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-kube-api-access-f5tzf\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.980657 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.980196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:18.993144 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.993113    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-mlbvr" event={"ID":"94714598-e34a-4953-b07d-c1b9037a561e","Type":"ContainerStarted","Data":"3c7837def9a3b86a384d8185ae8e8e76ec5e3568ea3a9b79439b6e529e9947b6"}
Jul 28 17:18:18.993860 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.993835    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:18:18.995955 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.995896    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4" event={"ID":"1b24d701-1478-47b1-b8da-839a8fe0034a","Type":"ContainerStarted","Data":"a144b0d0e78d09518bdbc2c4523ab75217a30f4842ac07fa0433660c484514fb"}
Jul 28 17:18:18.999588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:18.999518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgpvf" event={"ID":"08dc8e97-c2e1-4238-b172-a40801fca1a8","Type":"ContainerStarted","Data":"8d8413f7543c2ed16b2581bd0dbab42a7e77a6cbcbeacee87d3d56518ecd06b8"}
Jul 28 17:18:19.006129 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:19.006068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gjl4f" event={"ID":"42b3b5e9-0f43-4c50-be37-a7c61ff99746","Type":"ContainerStarted","Data":"1b565b2d77e664d5613554723ab248ba066e950ac9612ffb3f6f339e1fe5daac"}
Jul 28 17:18:19.008575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:19.008511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlxg9" event={"ID":"53101351-b715-4d1c-9072-67c1165ede98","Type":"ContainerStarted","Data":"e50ea55f8030603203a529038046aa9f23643eb609ed997829108b4ac7471666"}
Jul 28 17:18:19.020997 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:19.020902    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerStarted","Data":"00653433931d1fd25099320a085b13556301876f1b12346f2bc178d3ba6242a4"}
Jul 28 17:18:19.054910 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:19.053294    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-mlbvr" podStartSLOduration=66.324810069 podStartE2EDuration="1m15.053273781s" totalImagesPullingTime="8.728463712s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:19.028643166 +0000 UTC m=+75.962042589" watchObservedRunningTime="2026-07-28 17:18:19.053273781 +0000 UTC m=+75.986673204"
Jul 28 17:18:19.075575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:19.075194    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:19.331441 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:19.331375    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gjl4f" podStartSLOduration=66.588035931 podStartE2EDuration="1m15.331349819s" totalImagesPullingTime="8.743313888s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:19.054648261 +0000 UTC m=+75.988047680" watchObservedRunningTime="2026-07-28 17:18:19.331349819 +0000 UTC m=+76.264749242"
Jul 28 17:18:19.332922 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:19.332896    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:18:19.337746 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:18:19.337711    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903335c0_8a9c_4d1a_839f_c852baa6cca8.slice/crio-ecb216527e93ea977587ed3c86c711e011b3ac51539651f8c32dff8b943988d8 WatchSource:0}: Error finding container ecb216527e93ea977587ed3c86c711e011b3ac51539651f8c32dff8b943988d8: Status 404 returned error can't find the container with id ecb216527e93ea977587ed3c86c711e011b3ac51539651f8c32dff8b943988d8
Jul 28 17:18:20.029472 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:20.029435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerStarted","Data":"ecb216527e93ea977587ed3c86c711e011b3ac51539651f8c32dff8b943988d8"}
Jul 28 17:18:23.045685 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:23.045631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlxg9" event={"ID":"53101351-b715-4d1c-9072-67c1165ede98","Type":"ContainerStarted","Data":"789db2b8f19d23cf551dc47f14707e2281860f49c7e3039c0119312296e86e9e"}
Jul 28 17:18:23.047805 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:23.047779    2563 generic.go:356] "Generic (PLEG): container finished" podID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerID="c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3" exitCode=0
Jul 28 17:18:23.047897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:23.047881    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerDied","Data":"c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3"}
Jul 28 17:18:23.050730 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:23.050653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4" event={"ID":"1b24d701-1478-47b1-b8da-839a8fe0034a","Type":"ContainerStarted","Data":"edf48d03c07206be68d9dd00fab14dc253b360a0d1ed9921ca9350c650efa6c7"}
Jul 28 17:18:23.053796 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:23.053238    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgpvf" event={"ID":"08dc8e97-c2e1-4238-b172-a40801fca1a8","Type":"ContainerStarted","Data":"13757ae1ae6bc03e5bb4fca7e40da8e23558ffaaeb35c0a8213550403ee51115"}
Jul 28 17:18:23.056332 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:23.056086    2563 generic.go:356] "Generic (PLEG): container finished" podID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f" exitCode=0
Jul 28 17:18:23.056332 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:23.056128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerDied","Data":"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"}
Jul 28 17:18:23.116964 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:23.116917    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4" podStartSLOduration=2.818266161 podStartE2EDuration="7.11689865s" totalImagesPullingTime="4.298632489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:23.116243555 +0000 UTC m=+80.049642974" watchObservedRunningTime="2026-07-28 17:18:23.11689865 +0000 UTC m=+80.050298074"
Jul 28 17:18:24.063158 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:24.063113    2563 generic.go:356] "Generic (PLEG): container finished" podID="08dc8e97-c2e1-4238-b172-a40801fca1a8" containerID="13757ae1ae6bc03e5bb4fca7e40da8e23558ffaaeb35c0a8213550403ee51115" exitCode=0
Jul 28 17:18:24.063770 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:24.063735    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgpvf" event={"ID":"08dc8e97-c2e1-4238-b172-a40801fca1a8","Type":"ContainerDied","Data":"13757ae1ae6bc03e5bb4fca7e40da8e23558ffaaeb35c0a8213550403ee51115"}
Jul 28 17:18:25.071505 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:25.071466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgpvf" event={"ID":"08dc8e97-c2e1-4238-b172-a40801fca1a8","Type":"ContainerStarted","Data":"5ad3e374df45c303969d1b6677822303905ef43ab8d5611611a0d4c90d764e65"}
Jul 28 17:18:25.071973 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:25.071538    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-fgpvf" event={"ID":"08dc8e97-c2e1-4238-b172-a40801fca1a8","Type":"ContainerStarted","Data":"fbfeb4330ed5922de2cf37325f59250641608d2f9ca31af519a7280139f73099"}
Jul 28 17:18:25.096381 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:25.096313    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-fgpvf" podStartSLOduration=8.629410702 podStartE2EDuration="13.096291983s" totalImagesPullingTime="4.382157907s" totalInitContainerRuntime="84.723374ms" isStatefulPod=true podCreationTimestamp="2026-07-28 17:18:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:25.096158947 +0000 UTC m=+82.029558365" watchObservedRunningTime="2026-07-28 17:18:25.096291983 +0000 UTC m=+82.029691382"
Jul 28 17:18:28.559664 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:28.559518    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw" podUID="bdbf6db7-c494-4ec0-983c-5e90bd957f9c" containerName="registry" containerID="cri-o://65eb26f5da5174845a475616f3ab4f3faa66b34861194fe47da99c2c49e6b8f7" gracePeriod=30
Jul 28 17:18:32.100156 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:32.100119    2563 generic.go:356] "Generic (PLEG): container finished" podID="bdbf6db7-c494-4ec0-983c-5e90bd957f9c" containerID="65eb26f5da5174845a475616f3ab4f3faa66b34861194fe47da99c2c49e6b8f7" exitCode=0
Jul 28 17:18:32.100553 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:32.100210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw" event={"ID":"bdbf6db7-c494-4ec0-983c-5e90bd957f9c","Type":"ContainerDied","Data":"65eb26f5da5174845a475616f3ab4f3faa66b34861194fe47da99c2c49e6b8f7"}
Jul 28 17:18:36.035632 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.035601    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:18:36.107489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-certificates\") pod \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") "
Jul 28 17:18:36.107489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107492    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2969t\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-kube-api-access-2969t\") pod \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") "
Jul 28 17:18:36.107711 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107519    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") pod \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") "
Jul 28 17:18:36.107711 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107634    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-ca-trust-extracted\") pod \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") "
Jul 28 17:18:36.107711 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107689    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-bound-sa-token\") pod \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") "
Jul 28 17:18:36.107871 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107739    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-trusted-ca\") pod \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") "
Jul 28 17:18:36.107871 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107769    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-installation-pull-secrets\") pod \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") "
Jul 28 17:18:36.107871 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107800    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-image-registry-private-configuration\") pod \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\" (UID: \"bdbf6db7-c494-4ec0-983c-5e90bd957f9c\") "
Jul 28 17:18:36.108027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.107972    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "bdbf6db7-c494-4ec0-983c-5e90bd957f9c" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:36.108104 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.108084    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-certificates\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:18:36.108409 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.108377    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "bdbf6db7-c494-4ec0-983c-5e90bd957f9c" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:18:36.110058 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.109994    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "bdbf6db7-c494-4ec0-983c-5e90bd957f9c" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:36.110174 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.110156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "bdbf6db7-c494-4ec0-983c-5e90bd957f9c" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:36.110238 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.110216    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "bdbf6db7-c494-4ec0-983c-5e90bd957f9c" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:36.110294 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.110261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-kube-api-access-2969t" (OuterVolumeSpecName: "kube-api-access-2969t") pod "bdbf6db7-c494-4ec0-983c-5e90bd957f9c" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c"). InnerVolumeSpecName "kube-api-access-2969t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:18:36.110294 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.110266    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "bdbf6db7-c494-4ec0-983c-5e90bd957f9c" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:18:36.116990 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.116966    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "bdbf6db7-c494-4ec0-983c-5e90bd957f9c" (UID: "bdbf6db7-c494-4ec0-983c-5e90bd957f9c"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:18:36.117289 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.117267    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw" event={"ID":"bdbf6db7-c494-4ec0-983c-5e90bd957f9c","Type":"ContainerDied","Data":"6d9b555a145635611dfc500525ee1f71285ff80ac7b7fcf2f935dd6f62e946a3"}
Jul 28 17:18:36.117354 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.117326    2563 scope.go:117] "RemoveContainer" containerID="65eb26f5da5174845a475616f3ab4f3faa66b34861194fe47da99c2c49e6b8f7"
Jul 28 17:18:36.117354 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.117269    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw"
Jul 28 17:18:36.140157 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.140136    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77b5b5d789-5mprw"]
Jul 28 17:18:36.146687 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.145834    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77b5b5d789-5mprw"]
Jul 28 17:18:36.209096 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.209072    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-image-registry-private-configuration\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:18:36.209096 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.209095    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2969t\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-kube-api-access-2969t\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:18:36.209211 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.209106    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-registry-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:18:36.209211 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.209115    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-ca-trust-extracted\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:18:36.209211 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.209123    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-bound-sa-token\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:18:36.209211 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.209131    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-trusted-ca\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:18:36.209211 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:36.209139    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/bdbf6db7-c494-4ec0-983c-5e90bd957f9c-installation-pull-secrets\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:18:37.109837 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.109732    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:37.109837 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.109836    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:37.126828 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.126661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerStarted","Data":"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"}
Jul 28 17:18:37.126828 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.126767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerStarted","Data":"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"}
Jul 28 17:18:37.130507 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.130462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-7c97dc4555-sp2kw" event={"ID":"da8735fd-e062-458c-933f-7d344a34f479","Type":"ContainerStarted","Data":"54fea32f681759b0114745e1d66712c2da6bf990183c2efef1a66d13a84fdb3e"}
Jul 28 17:18:37.131017 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.130973    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-7c97dc4555-sp2kw"
Jul 28 17:18:37.135905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.135878    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qlxg9" event={"ID":"53101351-b715-4d1c-9072-67c1165ede98","Type":"ContainerStarted","Data":"16cd1728e48845868a0d2ce69d4b8818e26a9d0eab32d13e06ceedf22b8c10f1"}
Jul 28 17:18:37.140420 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.140394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerStarted","Data":"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f"}
Jul 28 17:18:37.140582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.140461    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerStarted","Data":"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a"}
Jul 28 17:18:37.140582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.140501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerStarted","Data":"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a"}
Jul 28 17:18:37.140582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.140536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerStarted","Data":"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69"}
Jul 28 17:18:37.140582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.140572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerStarted","Data":"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5"}
Jul 28 17:18:37.145281 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.145260    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-7c97dc4555-sp2kw"
Jul 28 17:18:37.149995 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.149949    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-7c97dc4555-sp2kw" podStartSLOduration=1.494411294 podStartE2EDuration="26.149932895s" totalImagesPullingTime="24.655521601s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:37.149838917 +0000 UTC m=+94.083238358" watchObservedRunningTime="2026-07-28 17:18:37.149932895 +0000 UTC m=+94.083332316"
Jul 28 17:18:37.175880 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.175838    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qlxg9" podStartSLOduration=34.265360383 podStartE2EDuration="59.175821501s" totalImagesPullingTime="24.910461118s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:17:38 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:37.174802818 +0000 UTC m=+94.108202258" watchObservedRunningTime="2026-07-28 17:18:37.175821501 +0000 UTC m=+94.109220920"
Jul 28 17:18:37.599700 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:37.599342    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdbf6db7-c494-4ec0-983c-5e90bd957f9c" path="/var/lib/kubelet/pods/bdbf6db7-c494-4ec0-983c-5e90bd957f9c/volumes"
Jul 28 17:18:38.542202 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:38.542167    2563 patch_prober.go:28] interesting pod/image-registry-77b5b5d789-5mprw container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.133.0.6:5000/healthz\": context deadline exceeded" start-of-body=
Jul 28 17:18:38.542650 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:38.542223    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-77b5b5d789-5mprw" podUID="bdbf6db7-c494-4ec0-983c-5e90bd957f9c" containerName="registry" probeResult="failure" output="Get \"https://10.133.0.6:5000/healthz\": context deadline exceeded"
Jul 28 17:18:40.158184 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:40.158104    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerStarted","Data":"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77"}
Jul 28 17:18:40.161969 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:40.161941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerStarted","Data":"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"}
Jul 28 17:18:40.162103 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:40.162008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerStarted","Data":"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"}
Jul 28 17:18:40.162103 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:40.162044    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerStarted","Data":"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"}
Jul 28 17:18:40.162103 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:40.162078    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerStarted","Data":"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"}
Jul 28 17:18:40.212478 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:40.212426    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=7.157169015 podStartE2EDuration="27.212406978s" totalImagesPullingTime="20.022487037s" totalInitContainerRuntime="32.750926ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:40.210803413 +0000 UTC m=+97.144202855" watchObservedRunningTime="2026-07-28 17:18:40.212406978 +0000 UTC m=+97.145806398"
Jul 28 17:18:40.256621 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:40.256565    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.322419399 podStartE2EDuration="22.256544568s" totalImagesPullingTime="19.902233165s" totalInitContainerRuntime="31.892004ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:18:18 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:18:40.254533579 +0000 UTC m=+97.187933034" watchObservedRunningTime="2026-07-28 17:18:40.256544568 +0000 UTC m=+97.189943988"
Jul 28 17:18:44.076068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:44.076030    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:18:51.035848 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:51.035816    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-mlbvr"
Jul 28 17:18:57.114109 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:57.114079    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:18:57.117807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:18:57.117787    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-77fcdc95f7-dg5d4"
Jul 28 17:19:19.076059 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:19.076022    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:19.147135 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:19.147110    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:19.323689 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:19.323646    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:32.776519 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:32.776482    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:19:32.777013 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:32.776791    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="alertmanager" containerID="cri-o://f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5" gracePeriod=120
Jul 28 17:19:32.777013 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:32.776871    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy-metric" containerID="cri-o://e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f" gracePeriod=120
Jul 28 17:19:32.777013 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:32.776920    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy" containerID="cri-o://2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a" gracePeriod=120
Jul 28 17:19:32.777013 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:32.776953    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="config-reloader" containerID="cri-o://06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69" gracePeriod=120
Jul 28 17:19:32.777013 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:32.776891    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy-web" containerID="cri-o://9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a" gracePeriod=120
Jul 28 17:19:32.777279 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:32.776964    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="prom-label-proxy" containerID="cri-o://3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77" gracePeriod=120
Jul 28 17:19:33.358397 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:33.358367    2563 generic.go:356] "Generic (PLEG): container finished" podID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerID="3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77" exitCode=0
Jul 28 17:19:33.358397 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:33.358391    2563 generic.go:356] "Generic (PLEG): container finished" podID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerID="2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a" exitCode=0
Jul 28 17:19:33.358397 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:33.358397    2563 generic.go:356] "Generic (PLEG): container finished" podID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerID="06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69" exitCode=0
Jul 28 17:19:33.358397 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:33.358404    2563 generic.go:356] "Generic (PLEG): container finished" podID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerID="f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5" exitCode=0
Jul 28 17:19:33.358652 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:33.358449    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerDied","Data":"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77"}
Jul 28 17:19:33.358652 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:33.358513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerDied","Data":"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a"}
Jul 28 17:19:33.358652 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:33.358544    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerDied","Data":"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69"}
Jul 28 17:19:33.358652 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:33.358569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerDied","Data":"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5"}
Jul 28 17:19:34.008937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.008913    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.159298 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-cluster-tls-config\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159298 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159263    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-trusted-ca-bundle\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159298 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159282    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-volume\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159304    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-metrics-client-ca\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159338    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-tls-assets\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159369    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-metric\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159396    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-out\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-web\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159483    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ttdh7\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-kube-api-access-ttdh7\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159537    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-main-tls\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159956 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159577    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-web-config\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159956 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-main-db\") pod \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\" (UID: \"cffed101-7b2e-4bd3-adc0-17bb1f0bb188\") "
Jul 28 17:19:34.159956 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.159725    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:34.160121 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.160021    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.160121 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.160028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:19:34.160522 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.160345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:34.162765 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.162730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-volume" (OuterVolumeSpecName: "config-volume") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:34.162996 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.162959    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:34.163463 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.163430    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-kube-api-access-ttdh7" (OuterVolumeSpecName: "kube-api-access-ttdh7") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "kube-api-access-ttdh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:34.163566 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.163505    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:34.163633 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.163602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-out" (OuterVolumeSpecName: "config-out") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:19:34.163710 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.163635    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:34.163800 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.163774    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:34.163947 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.163931    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:34.166783 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.166767    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:34.172552 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.172534    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-web-config" (OuterVolumeSpecName: "web-config") pod "cffed101-7b2e-4bd3-adc0-17bb1f0bb188" (UID: "cffed101-7b2e-4bd3-adc0-17bb1f0bb188"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:34.260933 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260908    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.260933 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260930    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-out\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260940    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260950    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260959    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ttdh7\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-kube-api-access-ttdh7\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260969    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-secret-alertmanager-main-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260977    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-web-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260985    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-alertmanager-main-db\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.260993    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-cluster-tls-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.261001    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-config-volume\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.261009    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-metrics-client-ca\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.261049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.261017    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/cffed101-7b2e-4bd3-adc0-17bb1f0bb188-tls-assets\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:34.364799 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.364775    2563 generic.go:356] "Generic (PLEG): container finished" podID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerID="e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f" exitCode=0
Jul 28 17:19:34.364799 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.364800    2563 generic.go:356] "Generic (PLEG): container finished" podID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerID="9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a" exitCode=0
Jul 28 17:19:34.364932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.364863    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.364967 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.364857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerDied","Data":"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f"}
Jul 28 17:19:34.365027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.365014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerDied","Data":"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a"}
Jul 28 17:19:34.365061 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.365047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"cffed101-7b2e-4bd3-adc0-17bb1f0bb188","Type":"ContainerDied","Data":"00653433931d1fd25099320a085b13556301876f1b12346f2bc178d3ba6242a4"}
Jul 28 17:19:34.365093 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.365079    2563 scope.go:117] "RemoveContainer" containerID="3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77"
Jul 28 17:19:34.372830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.372811    2563 scope.go:117] "RemoveContainer" containerID="e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f"
Jul 28 17:19:34.379041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.379025    2563 scope.go:117] "RemoveContainer" containerID="2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a"
Jul 28 17:19:34.385344 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.385327    2563 scope.go:117] "RemoveContainer" containerID="9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a"
Jul 28 17:19:34.392662 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.392638    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:19:34.393026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.393004    2563 scope.go:117] "RemoveContainer" containerID="06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69"
Jul 28 17:19:34.396090 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.396067    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:19:34.400532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.400516    2563 scope.go:117] "RemoveContainer" containerID="f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5"
Jul 28 17:19:34.406892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.406877    2563 scope.go:117] "RemoveContainer" containerID="c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3"
Jul 28 17:19:34.412916 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.412902    2563 scope.go:117] "RemoveContainer" containerID="3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77"
Jul 28 17:19:34.413144 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:34.413126    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77\": container with ID starting with 3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77 not found: ID does not exist" containerID="3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77"
Jul 28 17:19:34.413199 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.413151    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77"} err="failed to get container status \"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77\": rpc error: code = NotFound desc = could not find container \"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77\": container with ID starting with 3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77 not found: ID does not exist"
Jul 28 17:19:34.413199 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.413182    2563 scope.go:117] "RemoveContainer" containerID="e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f"
Jul 28 17:19:34.413398 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:34.413382    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f\": container with ID starting with e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f not found: ID does not exist" containerID="e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f"
Jul 28 17:19:34.413446 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.413401    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f"} err="failed to get container status \"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f\": rpc error: code = NotFound desc = could not find container \"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f\": container with ID starting with e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f not found: ID does not exist"
Jul 28 17:19:34.413446 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.413414    2563 scope.go:117] "RemoveContainer" containerID="2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a"
Jul 28 17:19:34.413615 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:34.413601    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a\": container with ID starting with 2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a not found: ID does not exist" containerID="2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a"
Jul 28 17:19:34.413653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.413620    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a"} err="failed to get container status \"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a\": rpc error: code = NotFound desc = could not find container \"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a\": container with ID starting with 2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a not found: ID does not exist"
Jul 28 17:19:34.413653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.413632    2563 scope.go:117] "RemoveContainer" containerID="9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a"
Jul 28 17:19:34.413838 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:34.413824    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a\": container with ID starting with 9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a not found: ID does not exist" containerID="9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a"
Jul 28 17:19:34.413876 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.413840    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a"} err="failed to get container status \"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a\": rpc error: code = NotFound desc = could not find container \"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a\": container with ID starting with 9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a not found: ID does not exist"
Jul 28 17:19:34.413876 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.413850    2563 scope.go:117] "RemoveContainer" containerID="06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69"
Jul 28 17:19:34.414051 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:34.414036    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69\": container with ID starting with 06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69 not found: ID does not exist" containerID="06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69"
Jul 28 17:19:34.414090 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414054    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69"} err="failed to get container status \"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69\": rpc error: code = NotFound desc = could not find container \"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69\": container with ID starting with 06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69 not found: ID does not exist"
Jul 28 17:19:34.414090 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414064    2563 scope.go:117] "RemoveContainer" containerID="f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5"
Jul 28 17:19:34.414239 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:34.414223    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5\": container with ID starting with f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5 not found: ID does not exist" containerID="f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5"
Jul 28 17:19:34.414275 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414242    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5"} err="failed to get container status \"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5\": rpc error: code = NotFound desc = could not find container \"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5\": container with ID starting with f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5 not found: ID does not exist"
Jul 28 17:19:34.414275 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414253    2563 scope.go:117] "RemoveContainer" containerID="c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3"
Jul 28 17:19:34.414446 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:34.414432    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3\": container with ID starting with c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3 not found: ID does not exist" containerID="c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3"
Jul 28 17:19:34.414492 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414449    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3"} err="failed to get container status \"c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3\": rpc error: code = NotFound desc = could not find container \"c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3\": container with ID starting with c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3 not found: ID does not exist"
Jul 28 17:19:34.414492 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414461    2563 scope.go:117] "RemoveContainer" containerID="3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77"
Jul 28 17:19:34.414645 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414620    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77"} err="failed to get container status \"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77\": rpc error: code = NotFound desc = could not find container \"3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77\": container with ID starting with 3651948d87e1da04fc2f91a4087e6d3ecafbb602d9623413fc03d703909c7a77 not found: ID does not exist"
Jul 28 17:19:34.414700 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414645    2563 scope.go:117] "RemoveContainer" containerID="e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f"
Jul 28 17:19:34.414848 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414833    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f"} err="failed to get container status \"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f\": rpc error: code = NotFound desc = could not find container \"e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f\": container with ID starting with e7b2eff2e937700fc5dbaa1761140d73b6e71f66fa1c4190326dce2ef518f27f not found: ID does not exist"
Jul 28 17:19:34.414848 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.414847    2563 scope.go:117] "RemoveContainer" containerID="2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a"
Jul 28 17:19:34.415088 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415069    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a"} err="failed to get container status \"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a\": rpc error: code = NotFound desc = could not find container \"2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a\": container with ID starting with 2f836ab64330af9bfe942ddddb2bd62d1c3c03694e2465e01255bdde5222da3a not found: ID does not exist"
Jul 28 17:19:34.415130 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415090    2563 scope.go:117] "RemoveContainer" containerID="9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a"
Jul 28 17:19:34.415285 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415267    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a"} err="failed to get container status \"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a\": rpc error: code = NotFound desc = could not find container \"9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a\": container with ID starting with 9ab3f26b10a2c708dc88e6a0fb2b5b481229b40fed368fbe0fc7f6999925d71a not found: ID does not exist"
Jul 28 17:19:34.415326 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415285    2563 scope.go:117] "RemoveContainer" containerID="06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69"
Jul 28 17:19:34.415488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415473    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69"} err="failed to get container status \"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69\": rpc error: code = NotFound desc = could not find container \"06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69\": container with ID starting with 06d8e4d0766e48e93874f6e5de37325daba2f85bbe6e6ef51acea3a44f732b69 not found: ID does not exist"
Jul 28 17:19:34.415488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415487    2563 scope.go:117] "RemoveContainer" containerID="f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5"
Jul 28 17:19:34.415713 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415692    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5"} err="failed to get container status \"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5\": rpc error: code = NotFound desc = could not find container \"f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5\": container with ID starting with f458d4333508486df5b208762467b350cac1544ca8f6b80172373a888fe6e8d5 not found: ID does not exist"
Jul 28 17:19:34.415759 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415714    2563 scope.go:117] "RemoveContainer" containerID="c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3"
Jul 28 17:19:34.415909 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.415893    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3"} err="failed to get container status \"c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3\": rpc error: code = NotFound desc = could not find container \"c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3\": container with ID starting with c347b2e6268b8ccf4e994fde0f1894656f412aa9f40b910c5497c1208ca33bb3 not found: ID does not exist"
Jul 28 17:19:34.427625 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427604    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:19:34.427887 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427871    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="alertmanager"
Jul 28 17:19:34.427941 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427890    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="alertmanager"
Jul 28 17:19:34.427941 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427902    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="config-reloader"
Jul 28 17:19:34.427941 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427908    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="config-reloader"
Jul 28 17:19:34.427941 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427926    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy-web"
Jul 28 17:19:34.427941 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427933    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy-web"
Jul 28 17:19:34.427941 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427942    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="init-config-reloader"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427948    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="init-config-reloader"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427955    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy-metric"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427961    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy-metric"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427969    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="prom-label-proxy"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427974    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="prom-label-proxy"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427981    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdbf6db7-c494-4ec0-983c-5e90bd957f9c" containerName="registry"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427986    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdbf6db7-c494-4ec0-983c-5e90bd957f9c" containerName="registry"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427994    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.427999    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.428036    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy-metric"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.428044    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="alertmanager"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.428049    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.428054    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="config-reloader"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.428059    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="kube-rbac-proxy-web"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.428064    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" containerName="prom-label-proxy"
Jul 28 17:19:34.428108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.428069    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdbf6db7-c494-4ec0-983c-5e90bd957f9c" containerName="registry"
Jul 28 17:19:34.431326 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.431312    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.436142 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 28 17:19:34.436421 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436396    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 28 17:19:34.436596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 28 17:19:34.436596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436472    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 28 17:19:34.436596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436502    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 28 17:19:34.436596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436457    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 28 17:19:34.436845 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 28 17:19:34.436887 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436855    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 28 17:19:34.436932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.436836    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-ndcpq\""
Jul 28 17:19:34.442627 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.442610    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 28 17:19:34.450140 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.450121    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:19:34.562778 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.562745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgbxn\" (UniqueName: \"kubernetes.io/projected/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-kube-api-access-mgbxn\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.562778 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.562781    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.562971 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.562806    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.562971 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.562857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563036 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.562966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-web-config\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563036 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.563001    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563100 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.563050    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-config-out\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563100 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.563072    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563168 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.563102    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563168 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.563118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563168 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.563135    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563291 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.563193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.563291 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.563261    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.663945 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.663888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.663945 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.663919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.663945 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.663940    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgbxn\" (UniqueName: \"kubernetes.io/projected/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-kube-api-access-mgbxn\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664147 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.663956    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664147 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.663971    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664147 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664147 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-web-config\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664340 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664340 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-config-out\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664340 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664254    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664340 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664340 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664358    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.664606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.664389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.665236 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.665208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667139 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667139 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667051    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667139 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-tls-assets\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667304 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667304 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667174    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667304 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667269    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-web-config\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667444 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667347    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667444 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667423    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-config-out\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.667567 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.667550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.668846 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.668829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-config-volume\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.671926 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.671908    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgbxn\" (UniqueName: \"kubernetes.io/projected/5f77bfc2-677a-4fa6-baeb-4b13150d76c7-kube-api-access-mgbxn\") pod \"alertmanager-main-0\" (UID: \"5f77bfc2-677a-4fa6-baeb-4b13150d76c7\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.740648 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.740627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 28 17:19:34.872040 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:34.872015    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 28 17:19:34.872883 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:19:34.872858    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f77bfc2_677a_4fa6_baeb_4b13150d76c7.slice/crio-e5890ae34a0c3c01b69a97781c490cb571edb68e1af22cf804281579dbea6e1e WatchSource:0}: Error finding container e5890ae34a0c3c01b69a97781c490cb571edb68e1af22cf804281579dbea6e1e: Status 404 returned error can't find the container with id e5890ae34a0c3c01b69a97781c490cb571edb68e1af22cf804281579dbea6e1e
Jul 28 17:19:35.371070 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:35.371032    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f77bfc2-677a-4fa6-baeb-4b13150d76c7" containerID="cc90550e13fb512e0a802c98a10b96c420e466fc2b2de21c38cd12fe332a14fb" exitCode=0
Jul 28 17:19:35.371461 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:35.371109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f77bfc2-677a-4fa6-baeb-4b13150d76c7","Type":"ContainerDied","Data":"cc90550e13fb512e0a802c98a10b96c420e466fc2b2de21c38cd12fe332a14fb"}
Jul 28 17:19:35.371461 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:35.371169    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f77bfc2-677a-4fa6-baeb-4b13150d76c7","Type":"ContainerStarted","Data":"e5890ae34a0c3c01b69a97781c490cb571edb68e1af22cf804281579dbea6e1e"}
Jul 28 17:19:35.598007 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:35.597562    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cffed101-7b2e-4bd3-adc0-17bb1f0bb188" path="/var/lib/kubelet/pods/cffed101-7b2e-4bd3-adc0-17bb1f0bb188/volumes"
Jul 28 17:19:36.378192 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.378146    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f77bfc2-677a-4fa6-baeb-4b13150d76c7","Type":"ContainerStarted","Data":"623a1499172dd8e4815d79442f2fbe1b19d6e6f6c43dc63c7cafd185661d49e7"}
Jul 28 17:19:36.378566 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.378213    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f77bfc2-677a-4fa6-baeb-4b13150d76c7","Type":"ContainerStarted","Data":"356ea1e890873679a6f53305bddc0531247f891914d6f7c32c8c0f83f6c1ffa6"}
Jul 28 17:19:36.378566 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.378240    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f77bfc2-677a-4fa6-baeb-4b13150d76c7","Type":"ContainerStarted","Data":"dc732cc80e79d21c6551ec0726e37fbf1c35f9a5cee2f752ae20410da01441e1"}
Jul 28 17:19:36.378566 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.378264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f77bfc2-677a-4fa6-baeb-4b13150d76c7","Type":"ContainerStarted","Data":"b918b6fbf2e949111c1934900876ef48dd6c82c6dd9a45a8f226612bb9a8ac82"}
Jul 28 17:19:36.378566 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.378316    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f77bfc2-677a-4fa6-baeb-4b13150d76c7","Type":"ContainerStarted","Data":"defa149167956b5b727f465efb94b9c61da42aa7e176bbdecf3ab928b7943130"}
Jul 28 17:19:36.378566 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.378354    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"5f77bfc2-677a-4fa6-baeb-4b13150d76c7","Type":"ContainerStarted","Data":"1184a8639052ddb1117fb07e7d8080a4ef7c8994108647c18ad128b3a6382bc5"}
Jul 28 17:19:36.414759 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.414706    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.39699849 podStartE2EDuration="2.414691133s" totalImagesPullingTime="0s" totalInitContainerRuntime="17.692643ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:36.414561043 +0000 UTC m=+153.347960461" watchObservedRunningTime="2026-07-28 17:19:36.414691133 +0000 UTC m=+153.348090545"
Jul 28 17:19:36.809624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.809537    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"]
Jul 28 17:19:36.811857 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.811842    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:36.814913 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.814889    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-9n94t\""
Jul 28 17:19:36.815049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.814923    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 28 17:19:36.815049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.814935    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 28 17:19:36.815049 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.815042    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 28 17:19:36.815625 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.815385    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 28 17:19:36.815625 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.815538    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 28 17:19:36.820232 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.820210    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 28 17:19:36.826448 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.826423    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"]
Jul 28 17:19:36.982468 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.982434    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-federate-client-tls\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:36.982653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.982522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-telemeter-client-tls\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:36.982653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.982552    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-secret-telemeter-client\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:36.982653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.982627    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-serving-certs-ca-bundle\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:36.982858 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.982658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:36.982858 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.982735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-metrics-client-ca\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:36.982858 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.982796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wrtqb\" (UniqueName: \"kubernetes.io/projected/b544baa2-b772-4e08-88d0-b0a42bdb3070-kube-api-access-wrtqb\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:36.983014 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:36.982830    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.075802 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.075767    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:19:37.076203 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.076166    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="prometheus" containerID="cri-o://3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece" gracePeriod=600
Jul 28 17:19:37.076443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.076199    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="thanos-sidecar" containerID="cri-o://8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19" gracePeriod=600
Jul 28 17:19:37.076527 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.076203    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy" containerID="cri-o://f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419" gracePeriod=600
Jul 28 17:19:37.076582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.076200    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy-thanos" containerID="cri-o://ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c" gracePeriod=600
Jul 28 17:19:37.076652 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.076197    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy-web" containerID="cri-o://cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439" gracePeriod=600
Jul 28 17:19:37.076732 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.076250    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="config-reloader" containerID="cri-o://ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03" gracePeriod=600
Jul 28 17:19:37.086684 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.086496    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wrtqb\" (UniqueName: \"kubernetes.io/projected/b544baa2-b772-4e08-88d0-b0a42bdb3070-kube-api-access-wrtqb\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.086684 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.086545    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.086684 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.086581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-federate-client-tls\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.086684 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.086650    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-telemeter-client-tls\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.086920 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.086701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-secret-telemeter-client\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.086920 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.086735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-serving-certs-ca-bundle\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.086920 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.086769    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.086920 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.086798    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-metrics-client-ca\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.088198 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.087551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-metrics-client-ca\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.088788 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.088529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-telemeter-trusted-ca-bundle\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.089307 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.089267    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b544baa2-b772-4e08-88d0-b0a42bdb3070-serving-certs-ca-bundle\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.092407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.092382    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-secret-telemeter-client\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.092407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.092977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-federate-client-tls\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.092407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.093044    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.092407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.093529    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/b544baa2-b772-4e08-88d0-b0a42bdb3070-telemeter-client-tls\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.096572 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.096447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wrtqb\" (UniqueName: \"kubernetes.io/projected/b544baa2-b772-4e08-88d0-b0a42bdb3070-kube-api-access-wrtqb\") pod \"telemeter-client-57c4bcdbbb-wngtq\" (UID: \"b544baa2-b772-4e08-88d0-b0a42bdb3070\") " pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.122877 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.122853    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"
Jul 28 17:19:37.267229 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.266893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq"]
Jul 28 17:19:37.351760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.351739    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.384971 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.384946    2563 generic.go:356] "Generic (PLEG): container finished" podID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c" exitCode=0
Jul 28 17:19:37.384971 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.384968    2563 generic.go:356] "Generic (PLEG): container finished" podID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419" exitCode=0
Jul 28 17:19:37.384971 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.384976    2563 generic.go:356] "Generic (PLEG): container finished" podID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439" exitCode=0
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.384983    2563 generic.go:356] "Generic (PLEG): container finished" podID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19" exitCode=0
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.384992    2563 generic.go:356] "Generic (PLEG): container finished" podID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03" exitCode=0
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.384998    2563 generic.go:356] "Generic (PLEG): container finished" podID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece" exitCode=0
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385023    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385028    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerDied","Data":"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"}
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerDied","Data":"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"}
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385171    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerDied","Data":"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"}
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385196    2563 scope.go:117] "RemoveContainer" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerDied","Data":"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"}
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerDied","Data":"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"}
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerDied","Data":"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"}
Jul 28 17:19:37.385443 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.385337    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"903335c0-8a9c-4d1a-839f-c852baa6cca8","Type":"ContainerDied","Data":"ecb216527e93ea977587ed3c86c711e011b3ac51539651f8c32dff8b943988d8"}
Jul 28 17:19:37.386563 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.386503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq" event={"ID":"b544baa2-b772-4e08-88d0-b0a42bdb3070","Type":"ContainerStarted","Data":"65177f4b55394296dfb5eb696cd038c30445b56f0d42f23784f8e3629cd863e1"}
Jul 28 17:19:37.400031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.400009    2563 scope.go:117] "RemoveContainer" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"
Jul 28 17:19:37.410259 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.410242    2563 scope.go:117] "RemoveContainer" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"
Jul 28 17:19:37.421139 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.421111    2563 scope.go:117] "RemoveContainer" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"
Jul 28 17:19:37.428705 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.428659    2563 scope.go:117] "RemoveContainer" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"
Jul 28 17:19:37.435498 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.435473    2563 scope.go:117] "RemoveContainer" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"
Jul 28 17:19:37.443862 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.443844    2563 scope.go:117] "RemoveContainer" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"
Jul 28 17:19:37.451575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.451551    2563 scope.go:117] "RemoveContainer" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"
Jul 28 17:19:37.451859 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:37.451842    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": container with ID starting with ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c not found: ID does not exist" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"
Jul 28 17:19:37.451927 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.451869    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"} err="failed to get container status \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": rpc error: code = NotFound desc = could not find container \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": container with ID starting with ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c not found: ID does not exist"
Jul 28 17:19:37.451927 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.451887    2563 scope.go:117] "RemoveContainer" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"
Jul 28 17:19:37.452157 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:37.452132    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": container with ID starting with f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419 not found: ID does not exist" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"
Jul 28 17:19:37.452204 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.452162    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"} err="failed to get container status \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": rpc error: code = NotFound desc = could not find container \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": container with ID starting with f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419 not found: ID does not exist"
Jul 28 17:19:37.452204 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.452178    2563 scope.go:117] "RemoveContainer" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"
Jul 28 17:19:37.452465 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:37.452445    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": container with ID starting with cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439 not found: ID does not exist" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"
Jul 28 17:19:37.452525 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.452473    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"} err="failed to get container status \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": rpc error: code = NotFound desc = could not find container \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": container with ID starting with cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439 not found: ID does not exist"
Jul 28 17:19:37.452525 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.452486    2563 scope.go:117] "RemoveContainer" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"
Jul 28 17:19:37.452722 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:37.452703    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": container with ID starting with 8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19 not found: ID does not exist" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"
Jul 28 17:19:37.452812 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.452723    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"} err="failed to get container status \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": rpc error: code = NotFound desc = could not find container \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": container with ID starting with 8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19 not found: ID does not exist"
Jul 28 17:19:37.452812 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.452735    2563 scope.go:117] "RemoveContainer" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"
Jul 28 17:19:37.452975 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:37.452957    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": container with ID starting with ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03 not found: ID does not exist" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"
Jul 28 17:19:37.453018 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.452980    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"} err="failed to get container status \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": rpc error: code = NotFound desc = could not find container \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": container with ID starting with ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03 not found: ID does not exist"
Jul 28 17:19:37.453018 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.452997    2563 scope.go:117] "RemoveContainer" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"
Jul 28 17:19:37.453201 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:37.453187    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": container with ID starting with 3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece not found: ID does not exist" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"
Jul 28 17:19:37.453236 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.453203    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"} err="failed to get container status \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": rpc error: code = NotFound desc = could not find container \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": container with ID starting with 3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece not found: ID does not exist"
Jul 28 17:19:37.453236 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.453215    2563 scope.go:117] "RemoveContainer" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"
Jul 28 17:19:37.453401 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:19:37.453387    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": container with ID starting with d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f not found: ID does not exist" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"
Jul 28 17:19:37.453439 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.453405    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"} err="failed to get container status \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": rpc error: code = NotFound desc = could not find container \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": container with ID starting with d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f not found: ID does not exist"
Jul 28 17:19:37.453439 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.453418    2563 scope.go:117] "RemoveContainer" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"
Jul 28 17:19:37.453614 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.453598    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"} err="failed to get container status \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": rpc error: code = NotFound desc = could not find container \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": container with ID starting with ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c not found: ID does not exist"
Jul 28 17:19:37.453614 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.453613    2563 scope.go:117] "RemoveContainer" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"
Jul 28 17:19:37.453802 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.453787    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"} err="failed to get container status \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": rpc error: code = NotFound desc = could not find container \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": container with ID starting with f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419 not found: ID does not exist"
Jul 28 17:19:37.453849 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.453802    2563 scope.go:117] "RemoveContainer" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"
Jul 28 17:19:37.454028 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454010    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"} err="failed to get container status \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": rpc error: code = NotFound desc = could not find container \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": container with ID starting with cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439 not found: ID does not exist"
Jul 28 17:19:37.454081 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454031    2563 scope.go:117] "RemoveContainer" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"
Jul 28 17:19:37.454272 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454255    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"} err="failed to get container status \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": rpc error: code = NotFound desc = could not find container \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": container with ID starting with 8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19 not found: ID does not exist"
Jul 28 17:19:37.454320 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454272    2563 scope.go:117] "RemoveContainer" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"
Jul 28 17:19:37.454489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454472    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"} err="failed to get container status \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": rpc error: code = NotFound desc = could not find container \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": container with ID starting with ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03 not found: ID does not exist"
Jul 28 17:19:37.454527 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454490    2563 scope.go:117] "RemoveContainer" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"
Jul 28 17:19:37.454709 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454694    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"} err="failed to get container status \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": rpc error: code = NotFound desc = could not find container \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": container with ID starting with 3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece not found: ID does not exist"
Jul 28 17:19:37.454762 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454710    2563 scope.go:117] "RemoveContainer" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"
Jul 28 17:19:37.454933 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454913    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"} err="failed to get container status \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": rpc error: code = NotFound desc = could not find container \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": container with ID starting with d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f not found: ID does not exist"
Jul 28 17:19:37.455019 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.454933    2563 scope.go:117] "RemoveContainer" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"
Jul 28 17:19:37.455169 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455151    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"} err="failed to get container status \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": rpc error: code = NotFound desc = could not find container \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": container with ID starting with ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c not found: ID does not exist"
Jul 28 17:19:37.455233 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455172    2563 scope.go:117] "RemoveContainer" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"
Jul 28 17:19:37.455387 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455372    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"} err="failed to get container status \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": rpc error: code = NotFound desc = could not find container \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": container with ID starting with f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419 not found: ID does not exist"
Jul 28 17:19:37.455432 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455387    2563 scope.go:117] "RemoveContainer" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"
Jul 28 17:19:37.455581 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455563    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"} err="failed to get container status \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": rpc error: code = NotFound desc = could not find container \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": container with ID starting with cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439 not found: ID does not exist"
Jul 28 17:19:37.455649 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455582    2563 scope.go:117] "RemoveContainer" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"
Jul 28 17:19:37.455798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455783    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"} err="failed to get container status \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": rpc error: code = NotFound desc = could not find container \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": container with ID starting with 8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19 not found: ID does not exist"
Jul 28 17:19:37.455866 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455799    2563 scope.go:117] "RemoveContainer" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"
Jul 28 17:19:37.456027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.455998    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"} err="failed to get container status \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": rpc error: code = NotFound desc = could not find container \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": container with ID starting with ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03 not found: ID does not exist"
Jul 28 17:19:37.456027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456016    2563 scope.go:117] "RemoveContainer" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"
Jul 28 17:19:37.456231 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456215    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"} err="failed to get container status \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": rpc error: code = NotFound desc = could not find container \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": container with ID starting with 3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece not found: ID does not exist"
Jul 28 17:19:37.456298 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456232    2563 scope.go:117] "RemoveContainer" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"
Jul 28 17:19:37.456442 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456424    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"} err="failed to get container status \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": rpc error: code = NotFound desc = could not find container \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": container with ID starting with d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f not found: ID does not exist"
Jul 28 17:19:37.456504 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456443    2563 scope.go:117] "RemoveContainer" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"
Jul 28 17:19:37.456641 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456625    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"} err="failed to get container status \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": rpc error: code = NotFound desc = could not find container \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": container with ID starting with ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c not found: ID does not exist"
Jul 28 17:19:37.456766 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456642    2563 scope.go:117] "RemoveContainer" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"
Jul 28 17:19:37.456901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456876    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"} err="failed to get container status \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": rpc error: code = NotFound desc = could not find container \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": container with ID starting with f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419 not found: ID does not exist"
Jul 28 17:19:37.456972 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.456903    2563 scope.go:117] "RemoveContainer" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"
Jul 28 17:19:37.457134 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.457117    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"} err="failed to get container status \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": rpc error: code = NotFound desc = could not find container \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": container with ID starting with cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439 not found: ID does not exist"
Jul 28 17:19:37.457193 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.457135    2563 scope.go:117] "RemoveContainer" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"
Jul 28 17:19:37.457374 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.457342    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"} err="failed to get container status \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": rpc error: code = NotFound desc = could not find container \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": container with ID starting with 8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19 not found: ID does not exist"
Jul 28 17:19:37.457374 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.457363    2563 scope.go:117] "RemoveContainer" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"
Jul 28 17:19:37.457584 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.457567    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"} err="failed to get container status \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": rpc error: code = NotFound desc = could not find container \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": container with ID starting with ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03 not found: ID does not exist"
Jul 28 17:19:37.457632 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.457586    2563 scope.go:117] "RemoveContainer" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"
Jul 28 17:19:37.457830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.457811    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"} err="failed to get container status \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": rpc error: code = NotFound desc = could not find container \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": container with ID starting with 3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece not found: ID does not exist"
Jul 28 17:19:37.457905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.457832    2563 scope.go:117] "RemoveContainer" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"
Jul 28 17:19:37.458052 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458035    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"} err="failed to get container status \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": rpc error: code = NotFound desc = could not find container \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": container with ID starting with d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f not found: ID does not exist"
Jul 28 17:19:37.458118 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458055    2563 scope.go:117] "RemoveContainer" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"
Jul 28 17:19:37.458276 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458257    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"} err="failed to get container status \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": rpc error: code = NotFound desc = could not find container \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": container with ID starting with ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c not found: ID does not exist"
Jul 28 17:19:37.458326 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458277    2563 scope.go:117] "RemoveContainer" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"
Jul 28 17:19:37.458482 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458466    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"} err="failed to get container status \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": rpc error: code = NotFound desc = could not find container \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": container with ID starting with f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419 not found: ID does not exist"
Jul 28 17:19:37.458525 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458483    2563 scope.go:117] "RemoveContainer" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"
Jul 28 17:19:37.458723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458702    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"} err="failed to get container status \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": rpc error: code = NotFound desc = could not find container \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": container with ID starting with cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439 not found: ID does not exist"
Jul 28 17:19:37.458783 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458724    2563 scope.go:117] "RemoveContainer" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"
Jul 28 17:19:37.458939 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458922    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"} err="failed to get container status \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": rpc error: code = NotFound desc = could not find container \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": container with ID starting with 8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19 not found: ID does not exist"
Jul 28 17:19:37.458979 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.458941    2563 scope.go:117] "RemoveContainer" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"
Jul 28 17:19:37.459143 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459127    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"} err="failed to get container status \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": rpc error: code = NotFound desc = could not find container \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": container with ID starting with ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03 not found: ID does not exist"
Jul 28 17:19:37.459181 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459143    2563 scope.go:117] "RemoveContainer" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"
Jul 28 17:19:37.459322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459307    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"} err="failed to get container status \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": rpc error: code = NotFound desc = could not find container \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": container with ID starting with 3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece not found: ID does not exist"
Jul 28 17:19:37.459366 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459323    2563 scope.go:117] "RemoveContainer" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"
Jul 28 17:19:37.459508 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459493    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"} err="failed to get container status \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": rpc error: code = NotFound desc = could not find container \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": container with ID starting with d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f not found: ID does not exist"
Jul 28 17:19:37.459545 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459508    2563 scope.go:117] "RemoveContainer" containerID="ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"
Jul 28 17:19:37.459723 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459702    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c"} err="failed to get container status \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": rpc error: code = NotFound desc = could not find container \"ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c\": container with ID starting with ebd708bd256147adfdeacde08787a9449cfc69c1d24bebd4bc6ad2ab38e6409c not found: ID does not exist"
Jul 28 17:19:37.459759 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459726    2563 scope.go:117] "RemoveContainer" containerID="f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"
Jul 28 17:19:37.459935 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459919    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419"} err="failed to get container status \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": rpc error: code = NotFound desc = could not find container \"f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419\": container with ID starting with f17a8e314e77efedcbc303ab16da8bbf2aee81ff33fe0618a85ecca1346f1419 not found: ID does not exist"
Jul 28 17:19:37.459982 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.459934    2563 scope.go:117] "RemoveContainer" containerID="cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"
Jul 28 17:19:37.460160 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460141    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439"} err="failed to get container status \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": rpc error: code = NotFound desc = could not find container \"cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439\": container with ID starting with cefe98d40d893735e49215255e7496723468ebab89cac8a9b61c5f2484bc2439 not found: ID does not exist"
Jul 28 17:19:37.460195 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460161    2563 scope.go:117] "RemoveContainer" containerID="8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"
Jul 28 17:19:37.460325 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460308    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19"} err="failed to get container status \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": rpc error: code = NotFound desc = could not find container \"8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19\": container with ID starting with 8dd6353d52c3f5fe938c8195e8c56858b9959b2c53f1a15a38c0b6e3698d6a19 not found: ID does not exist"
Jul 28 17:19:37.460375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460327    2563 scope.go:117] "RemoveContainer" containerID="ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"
Jul 28 17:19:37.460546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460531    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03"} err="failed to get container status \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": rpc error: code = NotFound desc = could not find container \"ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03\": container with ID starting with ec1351acfb0af462cfabbf4e0584f800959da27e8b6cedd512da755d32794d03 not found: ID does not exist"
Jul 28 17:19:37.460546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460545    2563 scope.go:117] "RemoveContainer" containerID="3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"
Jul 28 17:19:37.460753 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460736    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece"} err="failed to get container status \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": rpc error: code = NotFound desc = could not find container \"3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece\": container with ID starting with 3a02e31accbdc884706c17c2b13be634eb0b9ac64310c9bb0f1149eab4cafece not found: ID does not exist"
Jul 28 17:19:37.460800 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460754    2563 scope.go:117] "RemoveContainer" containerID="d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"
Jul 28 17:19:37.460986 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.460970    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f"} err="failed to get container status \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": rpc error: code = NotFound desc = could not find container \"d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f\": container with ID starting with d5fdd9bf27847e9409761c20eca4844ecefb7261fc8b07330afb98c6dde53a3f not found: ID does not exist"
Jul 28 17:19:37.497841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.497815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-kube-rbac-proxy\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.497938 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.497869    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-serving-certs-ca-bundle\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.497938 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.497916    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-trusted-ca-bundle\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498034 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.497951    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-web-config\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498034 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.497982    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-1\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498034 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498011    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5tzf\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-kube-api-access-f5tzf\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498158 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498038    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-db\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498158 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-metrics-client-certs\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498158 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498106    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-2\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498158 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498138    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-thanos-prometheus-http-client-file\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498317 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498170    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-tls\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498317 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498220    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-kubelet-serving-ca-bundle\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498317 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498256    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-metrics-client-ca\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498317 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498287    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-config-out\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498483 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498429    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-config\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498483 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498470    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498572 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-0\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498572 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498547    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498656 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498587    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-tls-assets\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.498656 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.498613    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-grpc-tls\") pod \"903335c0-8a9c-4d1a-839f-c852baa6cca8\" (UID: \"903335c0-8a9c-4d1a-839f-c852baa6cca8\") "
Jul 28 17:19:37.500439 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.500184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:37.500439 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.500325    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:37.501568 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.501374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:37.501568 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.501450    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-config-out" (OuterVolumeSpecName: "config-out") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:19:37.501833 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.501783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:37.502834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.502558    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:37.503352 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.503296    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:37.503352 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.503322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:19:37.503489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.503408    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.503489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.503474    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:37.503611 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.503523    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-config" (OuterVolumeSpecName: "config") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.505109 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.505074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.505923 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.505901    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.506015 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.505988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:19:37.506507 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.506463    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.506507 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.506466    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.506633 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.506618    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-kube-api-access-f5tzf" (OuterVolumeSpecName: "kube-api-access-f5tzf") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "kube-api-access-f5tzf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:19:37.507494 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.507469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.507641 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.507621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.515123 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.515102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-web-config" (OuterVolumeSpecName: "web-config") pod "903335c0-8a9c-4d1a-839f-c852baa6cca8" (UID: "903335c0-8a9c-4d1a-839f-c852baa6cca8"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:19:37.599462 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599410    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-web-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599462 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599429    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599462 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599439    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5tzf\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-kube-api-access-f5tzf\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599462 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599448    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-db\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599462 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599456    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-metrics-client-certs\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599462 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599464    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599473    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-thanos-prometheus-http-client-file\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599482    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599491    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599499    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-metrics-client-ca\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599508    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/903335c0-8a9c-4d1a-839f-c852baa6cca8-config-out\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599521    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599530    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599539    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599552    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599561    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/903335c0-8a9c-4d1a-839f-c852baa6cca8-tls-assets\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599569    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-grpc-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599576    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/903335c0-8a9c-4d1a-839f-c852baa6cca8-secret-kube-rbac-proxy\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599586    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.599807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.599598    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/903335c0-8a9c-4d1a-839f-c852baa6cca8-prometheus-trusted-ca-bundle\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:19:37.709779 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.709751    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:19:37.714708 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.714686    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:19:37.748419 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748388    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:19:37.748641 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748628    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="prometheus"
Jul 28 17:19:37.748706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748643    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="prometheus"
Jul 28 17:19:37.748706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748653    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="thanos-sidecar"
Jul 28 17:19:37.748706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748658    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="thanos-sidecar"
Jul 28 17:19:37.748706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748681    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="init-config-reloader"
Jul 28 17:19:37.748706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748687    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="init-config-reloader"
Jul 28 17:19:37.748706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748694    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy-web"
Jul 28 17:19:37.748706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748703    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy-web"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748713    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="config-reloader"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748718    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="config-reloader"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748724    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy-thanos"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748729    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy-thanos"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748739    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748744    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748790    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="prometheus"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748799    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="config-reloader"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748804    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="thanos-sidecar"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748810    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy-web"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748815    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy-thanos"
Jul 28 17:19:37.748892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.748821    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" containerName="kube-rbac-proxy"
Jul 28 17:19:37.752290 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.752273    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.756507 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 28 17:19:37.756603 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756544    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 28 17:19:37.756603 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756557    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 28 17:19:37.756734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756543    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 28 17:19:37.756734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756627    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 28 17:19:37.756734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756717    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 28 17:19:37.756943 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756927    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 28 17:19:37.756978 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756953    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 28 17:19:37.757016 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756986    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 28 17:19:37.757061 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.756955    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 28 17:19:37.757114 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.757066    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 28 17:19:37.757243 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.757213    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-48p8q\""
Jul 28 17:19:37.757439 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.757425    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 28 17:19:37.757587 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.757541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-euqfncq37bf1t\""
Jul 28 17:19:37.761223 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.761189    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 28 17:19:37.763441 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.763422    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 28 17:19:37.775347 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.775326    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:19:37.901539 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901539 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901505    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4da263b-494e-4aa0-b450-4fe5c26d4430-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901539 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4da263b-494e-4aa0-b450-4fe5c26d4430-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901762 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901584    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901762 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901762 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901646    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901762 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901689    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kjfh\" (UniqueName: \"kubernetes.io/projected/e4da263b-494e-4aa0-b450-4fe5c26d4430-kube-api-access-7kjfh\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901762 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901750    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901904 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901904 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901795    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901904 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901814    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901904 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901904 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901853    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901904 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901870    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.901904 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901888    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.902089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901926    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.902089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.902089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-config\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.902089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.901979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:37.902089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:37.902017    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003069 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003069 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003069    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003290 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003290 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003117    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003290 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003421 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003311    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003421 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003354    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003421 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003421 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003550 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003550 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003444    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-config\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003550 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003463    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003550 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003550 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003550 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003549    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4da263b-494e-4aa0-b450-4fe5c26d4430-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003827 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003567    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4da263b-494e-4aa0-b450-4fe5c26d4430-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003827 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003827 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.003827 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003744    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.004027 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.003870    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.004253 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.004230    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.004344 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.004286    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kjfh\" (UniqueName: \"kubernetes.io/projected/e4da263b-494e-4aa0-b450-4fe5c26d4430-kube-api-access-7kjfh\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.004988 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.004890    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.005601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.005252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.005601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.005278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.006208    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.006298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.006880    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4da263b-494e-4aa0-b450-4fe5c26d4430-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.007416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.008055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-config\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.008087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.008212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4da263b-494e-4aa0-b450-4fe5c26d4430-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.008682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.008663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.009024    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009934 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.009481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.009934 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.009911    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.010118 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.010094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4da263b-494e-4aa0-b450-4fe5c26d4430-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.010267 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.010246    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4da263b-494e-4aa0-b450-4fe5c26d4430-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.013218 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.013201    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kjfh\" (UniqueName: \"kubernetes.io/projected/e4da263b-494e-4aa0-b450-4fe5c26d4430-kube-api-access-7kjfh\") pod \"prometheus-k8s-0\" (UID: \"e4da263b-494e-4aa0-b450-4fe5c26d4430\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.063076 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.063053    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:19:38.208537 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.208514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 28 17:19:38.210395 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:19:38.210360    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4da263b_494e_4aa0_b450_4fe5c26d4430.slice/crio-cec09d65e406cf71a2d5af6a8fd5418017882e96e1172f0f720bbf1a47cda1a9 WatchSource:0}: Error finding container cec09d65e406cf71a2d5af6a8fd5418017882e96e1172f0f720bbf1a47cda1a9: Status 404 returned error can't find the container with id cec09d65e406cf71a2d5af6a8fd5418017882e96e1172f0f720bbf1a47cda1a9
Jul 28 17:19:38.393794 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.393763    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4da263b-494e-4aa0-b450-4fe5c26d4430" containerID="3aa637de84cf8b7f2ce141a5a60717deb75f84c931c55897eed07aeb41e07298" exitCode=0
Jul 28 17:19:38.394107 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.393852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4da263b-494e-4aa0-b450-4fe5c26d4430","Type":"ContainerDied","Data":"3aa637de84cf8b7f2ce141a5a60717deb75f84c931c55897eed07aeb41e07298"}
Jul 28 17:19:38.394107 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:38.393931    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4da263b-494e-4aa0-b450-4fe5c26d4430","Type":"ContainerStarted","Data":"cec09d65e406cf71a2d5af6a8fd5418017882e96e1172f0f720bbf1a47cda1a9"}
Jul 28 17:19:39.400162 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.400127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq" event={"ID":"b544baa2-b772-4e08-88d0-b0a42bdb3070","Type":"ContainerStarted","Data":"be490138cb67e1b33aacb252927906ce15ac6b65a062826ac0e35251740163c5"}
Jul 28 17:19:39.400601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.400230    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq" event={"ID":"b544baa2-b772-4e08-88d0-b0a42bdb3070","Type":"ContainerStarted","Data":"89aaf4b626cbccc1c85ac6b2dc253b353eddb2f798eefba8e03ae4ee6cc8576e"}
Jul 28 17:19:39.400601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.400264    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq" event={"ID":"b544baa2-b772-4e08-88d0-b0a42bdb3070","Type":"ContainerStarted","Data":"017063ca91c7221afd477e2efb52410f787e8249c539534b4de7a236312006fd"}
Jul 28 17:19:39.403408 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.403384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4da263b-494e-4aa0-b450-4fe5c26d4430","Type":"ContainerStarted","Data":"da96af7f798d8d8726fc6f25f9aedb639ed25ffbc56e7c6f856d722ae4023a08"}
Jul 28 17:19:39.403508 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.403451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4da263b-494e-4aa0-b450-4fe5c26d4430","Type":"ContainerStarted","Data":"c6ae8c7aa70751b2c5141fcf4bf1d207c906908318293321cdebfea0e8657e26"}
Jul 28 17:19:39.403508 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.403491    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4da263b-494e-4aa0-b450-4fe5c26d4430","Type":"ContainerStarted","Data":"411487e74ef8537328a3b974e5e7f1b78417ef69e0a7be0f92cc4e21131688b9"}
Jul 28 17:19:39.403609 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.403530    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4da263b-494e-4aa0-b450-4fe5c26d4430","Type":"ContainerStarted","Data":"b894dfa5e88bfdb47beb9aed380b882adc30ba4a0a5319a8c960adfa75642124"}
Jul 28 17:19:39.403609 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.403570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4da263b-494e-4aa0-b450-4fe5c26d4430","Type":"ContainerStarted","Data":"d929bb7a9007a54515d14ce6dbc5adfefafa2e85e2e85f504fc0b2074e2fc494"}
Jul 28 17:19:39.403609 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.403599    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4da263b-494e-4aa0-b450-4fe5c26d4430","Type":"ContainerStarted","Data":"d6ab412a5663f2a8dd446f50594bdfb17316baf9628672fc491ee52f4cf07f9f"}
Jul 28 17:19:39.427816 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.427757    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-57c4bcdbbb-wngtq" podStartSLOduration=1.647657117 podStartE2EDuration="3.427742857s" totalImagesPullingTime="1.78008574s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:39.426446125 +0000 UTC m=+156.359845544" watchObservedRunningTime="2026-07-28 17:19:39.427742857 +0000 UTC m=+156.361142276"
Jul 28 17:19:39.456683 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.456625    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.435517155 podStartE2EDuration="2.456608607s" totalImagesPullingTime="0s" totalInitContainerRuntime="21.091452ms" isStatefulPod=false podCreationTimestamp="2026-07-28 17:19:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:19:39.455248128 +0000 UTC m=+156.388647567" watchObservedRunningTime="2026-07-28 17:19:39.456608607 +0000 UTC m=+156.390008027"
Jul 28 17:19:39.596290 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:39.596264    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="903335c0-8a9c-4d1a-839f-c852baa6cca8" path="/var/lib/kubelet/pods/903335c0-8a9c-4d1a-839f-c852baa6cca8/volumes"
Jul 28 17:19:43.063441 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:19:43.063400    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:20:38.064240 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:20:38.064205    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:20:38.081280 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:20:38.081248    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:20:38.633522 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:20:38.633494    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 28 17:22:03.456761 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:22:03.456727    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 17:23:42.798799 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.798764    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-lkngb"]
Jul 28 17:23:42.801866 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.801849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:42.804643 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.804622    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-bd94r\""
Jul 28 17:23:42.804758 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.804630    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 17:23:42.805732 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.805718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 17:23:42.805788 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.805729    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 17:23:42.812769 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.812751    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-lkngb"]
Jul 28 17:23:42.928529 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.928503    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzhmt\" (UniqueName: \"kubernetes.io/projected/35278a5d-2d1b-419a-9004-4b2e8f9eb4ee-kube-api-access-rzhmt\") pod \"seaweedfs-64568bcd49-lkngb\" (UID: \"35278a5d-2d1b-419a-9004-4b2e8f9eb4ee\") " pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:42.928639 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:42.928539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35278a5d-2d1b-419a-9004-4b2e8f9eb4ee-data\") pod \"seaweedfs-64568bcd49-lkngb\" (UID: \"35278a5d-2d1b-419a-9004-4b2e8f9eb4ee\") " pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:43.029594 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:43.029569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzhmt\" (UniqueName: \"kubernetes.io/projected/35278a5d-2d1b-419a-9004-4b2e8f9eb4ee-kube-api-access-rzhmt\") pod \"seaweedfs-64568bcd49-lkngb\" (UID: \"35278a5d-2d1b-419a-9004-4b2e8f9eb4ee\") " pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:43.029722 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:43.029606    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35278a5d-2d1b-419a-9004-4b2e8f9eb4ee-data\") pod \"seaweedfs-64568bcd49-lkngb\" (UID: \"35278a5d-2d1b-419a-9004-4b2e8f9eb4ee\") " pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:43.029968 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:43.029949    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/35278a5d-2d1b-419a-9004-4b2e8f9eb4ee-data\") pod \"seaweedfs-64568bcd49-lkngb\" (UID: \"35278a5d-2d1b-419a-9004-4b2e8f9eb4ee\") " pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:43.039248 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:43.039232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzhmt\" (UniqueName: \"kubernetes.io/projected/35278a5d-2d1b-419a-9004-4b2e8f9eb4ee-kube-api-access-rzhmt\") pod \"seaweedfs-64568bcd49-lkngb\" (UID: \"35278a5d-2d1b-419a-9004-4b2e8f9eb4ee\") " pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:43.111205 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:43.111182    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:43.234415 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:43.234382    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-lkngb"]
Jul 28 17:23:43.237660 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:23:43.237633    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35278a5d_2d1b_419a_9004_4b2e8f9eb4ee.slice/crio-1c6053b8107c3f50145b14de644a1d2908472b22ed98579571e3f1093c5f3a29 WatchSource:0}: Error finding container 1c6053b8107c3f50145b14de644a1d2908472b22ed98579571e3f1093c5f3a29: Status 404 returned error can't find the container with id 1c6053b8107c3f50145b14de644a1d2908472b22ed98579571e3f1093c5f3a29
Jul 28 17:23:43.238894 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:43.238874    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:23:43.260206 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:43.260180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-lkngb" event={"ID":"35278a5d-2d1b-419a-9004-4b2e8f9eb4ee","Type":"ContainerStarted","Data":"1c6053b8107c3f50145b14de644a1d2908472b22ed98579571e3f1093c5f3a29"}
Jul 28 17:23:47.281528 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:47.281495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-lkngb" event={"ID":"35278a5d-2d1b-419a-9004-4b2e8f9eb4ee","Type":"ContainerStarted","Data":"943eb7230c5f3241378d25b1a982ef9beb0b781c1c00186767f7ba2bc319fd73"}
Jul 28 17:23:47.281912 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:47.281623    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:23:47.298243 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:47.298200    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-lkngb" podStartSLOduration=2.315032538 podStartE2EDuration="5.298184959s" totalImagesPullingTime="2.983152421s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:23:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:23:47.296993611 +0000 UTC m=+404.230393043" watchObservedRunningTime="2026-07-28 17:23:47.298184959 +0000 UTC m=+404.231584377"
Jul 28 17:23:53.287107 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:23:53.287071    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-lkngb"
Jul 28 17:25:00.641549 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.641514    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/model-serving-api-7b954ffc95-hr54x"]
Jul 28 17:25:00.644413 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.644397    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:00.647308 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.647287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-tls\""
Jul 28 17:25:00.647427 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.647308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"model-serving-api-dockercfg-zj4gv\""
Jul 28 17:25:00.658104 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.658082    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-hr54x"]
Jul 28 17:25:00.774752 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.774730    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-48wrr\" (UniqueName: \"kubernetes.io/projected/e2c585ba-1b55-4306-a634-ca96d8490579-kube-api-access-48wrr\") pod \"model-serving-api-7b954ffc95-hr54x\" (UID: \"e2c585ba-1b55-4306-a634-ca96d8490579\") " pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:00.774866 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.774772    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c585ba-1b55-4306-a634-ca96d8490579-tls-certs\") pod \"model-serving-api-7b954ffc95-hr54x\" (UID: \"e2c585ba-1b55-4306-a634-ca96d8490579\") " pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:00.875112 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.875071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c585ba-1b55-4306-a634-ca96d8490579-tls-certs\") pod \"model-serving-api-7b954ffc95-hr54x\" (UID: \"e2c585ba-1b55-4306-a634-ca96d8490579\") " pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:00.875284 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.875172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-48wrr\" (UniqueName: \"kubernetes.io/projected/e2c585ba-1b55-4306-a634-ca96d8490579-kube-api-access-48wrr\") pod \"model-serving-api-7b954ffc95-hr54x\" (UID: \"e2c585ba-1b55-4306-a634-ca96d8490579\") " pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:00.875284 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:25:00.875205    2563 secret.go:189] Couldn't get secret kserve/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 28 17:25:00.875284 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:25:00.875265    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e2c585ba-1b55-4306-a634-ca96d8490579-tls-certs podName:e2c585ba-1b55-4306-a634-ca96d8490579 nodeName:}" failed. No retries permitted until 2026-07-28 17:25:01.375249643 +0000 UTC m=+478.308649040 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/e2c585ba-1b55-4306-a634-ca96d8490579-tls-certs") pod "model-serving-api-7b954ffc95-hr54x" (UID: "e2c585ba-1b55-4306-a634-ca96d8490579") : secret "model-serving-api-tls" not found
Jul 28 17:25:00.887077 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:00.887053    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-48wrr\" (UniqueName: \"kubernetes.io/projected/e2c585ba-1b55-4306-a634-ca96d8490579-kube-api-access-48wrr\") pod \"model-serving-api-7b954ffc95-hr54x\" (UID: \"e2c585ba-1b55-4306-a634-ca96d8490579\") " pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:01.380120 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:01.380074    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c585ba-1b55-4306-a634-ca96d8490579-tls-certs\") pod \"model-serving-api-7b954ffc95-hr54x\" (UID: \"e2c585ba-1b55-4306-a634-ca96d8490579\") " pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:01.382475 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:01.382455    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e2c585ba-1b55-4306-a634-ca96d8490579-tls-certs\") pod \"model-serving-api-7b954ffc95-hr54x\" (UID: \"e2c585ba-1b55-4306-a634-ca96d8490579\") " pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:01.553930 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:01.553897    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:01.682161 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:01.682133    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/model-serving-api-7b954ffc95-hr54x"]
Jul 28 17:25:01.684590 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:25:01.684558    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2c585ba_1b55_4306_a634_ca96d8490579.slice/crio-c7c9b59ab093086e620439a6d5b22af7374ffeb576c010f2ac6b0414ba82418c WatchSource:0}: Error finding container c7c9b59ab093086e620439a6d5b22af7374ffeb576c010f2ac6b0414ba82418c: Status 404 returned error can't find the container with id c7c9b59ab093086e620439a6d5b22af7374ffeb576c010f2ac6b0414ba82418c
Jul 28 17:25:02.559076 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:02.559035    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-hr54x" event={"ID":"e2c585ba-1b55-4306-a634-ca96d8490579","Type":"ContainerStarted","Data":"c7c9b59ab093086e620439a6d5b22af7374ffeb576c010f2ac6b0414ba82418c"}
Jul 28 17:25:04.568905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:04.568865    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/model-serving-api-7b954ffc95-hr54x" event={"ID":"e2c585ba-1b55-4306-a634-ca96d8490579","Type":"ContainerStarted","Data":"46e2ed1af46b5ebd445b4fe6bf61859424ca20e8a41b7e64f4159a9ad303a885"}
Jul 28 17:25:04.569279 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:04.568994    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:04.595665 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:04.595621    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/model-serving-api-7b954ffc95-hr54x" podStartSLOduration=2.216761397 podStartE2EDuration="4.595607732s" totalImagesPullingTime="2.378846335s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:04.594327713 +0000 UTC m=+481.527727148" watchObservedRunningTime="2026-07-28 17:25:04.595607732 +0000 UTC m=+481.529007150"
Jul 28 17:25:15.576853 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:15.576823    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/model-serving-api-7b954ffc95-hr54x"
Jul 28 17:25:17.520816 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.520778    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-f2899"]
Jul 28 17:25:17.523338 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.523322    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f2899"
Jul 28 17:25:17.536695 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.536658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-f2899"]
Jul 28 17:25:17.609591 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.609564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fngmd\" (UniqueName: \"kubernetes.io/projected/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-kube-api-access-fngmd\") pod \"s3-init-f2899\" (UID: \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\") " pod="kserve/s3-init-f2899"
Jul 28 17:25:17.609732 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.609653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-model-cache\") pod \"s3-init-f2899\" (UID: \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\") " pod="kserve/s3-init-f2899"
Jul 28 17:25:17.710031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.710003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-model-cache\") pod \"s3-init-f2899\" (UID: \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\") " pod="kserve/s3-init-f2899"
Jul 28 17:25:17.710148 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.710044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fngmd\" (UniqueName: \"kubernetes.io/projected/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-kube-api-access-fngmd\") pod \"s3-init-f2899\" (UID: \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\") " pod="kserve/s3-init-f2899"
Jul 28 17:25:17.710379 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.710361    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-model-cache\") pod \"s3-init-f2899\" (UID: \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\") " pod="kserve/s3-init-f2899"
Jul 28 17:25:17.718815 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.718792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fngmd\" (UniqueName: \"kubernetes.io/projected/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-kube-api-access-fngmd\") pod \"s3-init-f2899\" (UID: \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\") " pod="kserve/s3-init-f2899"
Jul 28 17:25:17.843604 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.843583    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f2899"
Jul 28 17:25:17.966542 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:17.966514    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-f2899"]
Jul 28 17:25:17.969403 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:25:17.969373    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podce51ebc8_0f9c_41ca_a52e_45a44f34928d.slice/crio-ad6a7bc32504e8f7de0aa9cdcd699937a4d723a83174b57e0c4457b1a5c8a876 WatchSource:0}: Error finding container ad6a7bc32504e8f7de0aa9cdcd699937a4d723a83174b57e0c4457b1a5c8a876: Status 404 returned error can't find the container with id ad6a7bc32504e8f7de0aa9cdcd699937a4d723a83174b57e0c4457b1a5c8a876
Jul 28 17:25:18.623212 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:18.623175    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f2899" event={"ID":"ce51ebc8-0f9c-41ca-a52e-45a44f34928d","Type":"ContainerStarted","Data":"ad6a7bc32504e8f7de0aa9cdcd699937a4d723a83174b57e0c4457b1a5c8a876"}
Jul 28 17:25:21.638201 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:21.638161    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f2899" event={"ID":"ce51ebc8-0f9c-41ca-a52e-45a44f34928d","Type":"ContainerStarted","Data":"258996c60e05bbd681cb93bf4ef89506fee5c58f26dcbd8d09a402d6bf3f18da"}
Jul 28 17:25:28.665118 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:28.665082    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce51ebc8-0f9c-41ca-a52e-45a44f34928d" containerID="258996c60e05bbd681cb93bf4ef89506fee5c58f26dcbd8d09a402d6bf3f18da" exitCode=0
Jul 28 17:25:28.665472 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:28.665155    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f2899" event={"ID":"ce51ebc8-0f9c-41ca-a52e-45a44f34928d","Type":"ContainerDied","Data":"258996c60e05bbd681cb93bf4ef89506fee5c58f26dcbd8d09a402d6bf3f18da"}
Jul 28 17:25:33.688490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:33.688464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f2899" event={"ID":"ce51ebc8-0f9c-41ca-a52e-45a44f34928d","Type":"ContainerStarted","Data":"0ebc1b4c86d43980ac0e10028983bfb34068b7bda6650289f0d0c9b7c7328884"}
Jul 28 17:25:34.710229 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:25:34.710180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-f2899" podStartSLOduration=2.259259954 podStartE2EDuration="17.710165651s" totalImagesPullingTime="8.180697299s" totalInitContainerRuntime="7.270208398s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:25:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:25:34.70932702 +0000 UTC m=+511.642726439" watchObservedRunningTime="2026-07-28 17:25:34.710165651 +0000 UTC m=+511.643565068"
Jul 28 17:26:38.931248 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:38.931214    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce51ebc8-0f9c-41ca-a52e-45a44f34928d" containerID="0ebc1b4c86d43980ac0e10028983bfb34068b7bda6650289f0d0c9b7c7328884" exitCode=0
Jul 28 17:26:38.931644 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:38.931289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f2899" event={"ID":"ce51ebc8-0f9c-41ca-a52e-45a44f34928d","Type":"ContainerDied","Data":"0ebc1b4c86d43980ac0e10028983bfb34068b7bda6650289f0d0c9b7c7328884"}
Jul 28 17:26:40.059373 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.059350    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f2899"
Jul 28 17:26:40.216269 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.216193    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-model-cache\") pod \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\" (UID: \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\") "
Jul 28 17:26:40.216269 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.216266    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fngmd\" (UniqueName: \"kubernetes.io/projected/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-kube-api-access-fngmd\") pod \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\" (UID: \"ce51ebc8-0f9c-41ca-a52e-45a44f34928d\") "
Jul 28 17:26:40.218299 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.218271    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-kube-api-access-fngmd" (OuterVolumeSpecName: "kube-api-access-fngmd") pod "ce51ebc8-0f9c-41ca-a52e-45a44f34928d" (UID: "ce51ebc8-0f9c-41ca-a52e-45a44f34928d"). InnerVolumeSpecName "kube-api-access-fngmd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:26:40.237921 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.237899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-model-cache" (OuterVolumeSpecName: "model-cache") pod "ce51ebc8-0f9c-41ca-a52e-45a44f34928d" (UID: "ce51ebc8-0f9c-41ca-a52e-45a44f34928d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:26:40.316846 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.316824    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fngmd\" (UniqueName: \"kubernetes.io/projected/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-kube-api-access-fngmd\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:26:40.316846 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.316845    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ce51ebc8-0f9c-41ca-a52e-45a44f34928d-model-cache\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:26:40.940316 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.940288    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-f2899"
Jul 28 17:26:40.940488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.940296    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-f2899" event={"ID":"ce51ebc8-0f9c-41ca-a52e-45a44f34928d","Type":"ContainerDied","Data":"ad6a7bc32504e8f7de0aa9cdcd699937a4d723a83174b57e0c4457b1a5c8a876"}
Jul 28 17:26:40.940488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:40.940401    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ad6a7bc32504e8f7de0aa9cdcd699937a4d723a83174b57e0c4457b1a5c8a876"
Jul 28 17:26:52.806760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.806723    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"]
Jul 28 17:26:52.807108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.807046    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce51ebc8-0f9c-41ca-a52e-45a44f34928d" containerName="s3-init"
Jul 28 17:26:52.807108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.807057    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce51ebc8-0f9c-41ca-a52e-45a44f34928d" containerName="s3-init"
Jul 28 17:26:52.807108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.807066    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce51ebc8-0f9c-41ca-a52e-45a44f34928d" containerName="download-hf-model"
Jul 28 17:26:52.807108 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.807073    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce51ebc8-0f9c-41ca-a52e-45a44f34928d" containerName="download-hf-model"
Jul 28 17:26:52.807239 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.807114    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce51ebc8-0f9c-41ca-a52e-45a44f34928d" containerName="s3-init"
Jul 28 17:26:52.811437 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.811422    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:52.814252 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.814223    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b19ff-predictor-serving-cert\""
Jul 28 17:26:52.814252 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.814251    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jwrfk\""
Jul 28 17:26:52.814421 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.814361    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:26:52.814694 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.814662    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:26:52.815429 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.815414    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:26:52.815536 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.815419    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-raw-sklearn-batcher-b19ff-kube-rbac-proxy-sar-config\""
Jul 28 17:26:52.815536 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.815494    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:26:52.821186 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.821166    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"]
Jul 28 17:26:52.905981 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.905954    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:52.906115 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.905994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42416c69-5294-432a-b19b-c77544078842-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:52.906115 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.906040    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-swrbf\" (UniqueName: \"kubernetes.io/projected/42416c69-5294-432a-b19b-c77544078842-kube-api-access-swrbf\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:52.906115 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.906086    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:52.906254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:52.906133    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.006855 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.006824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.006980 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.006867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42416c69-5294-432a-b19b-c77544078842-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.006980 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.006912    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-swrbf\" (UniqueName: \"kubernetes.io/projected/42416c69-5294-432a-b19b-c77544078842-kube-api-access-swrbf\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.006980 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.006960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.007136 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.006993    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.007196 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:26:53.007139    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-serving-cert: secret "isvc-raw-sklearn-batcher-b19ff-predictor-serving-cert" not found
Jul 28 17:26:53.007252 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:26:53.007219    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls podName:42416c69-5294-432a-b19b-c77544078842 nodeName:}" failed. No retries permitted until 2026-07-28 17:26:53.507199541 +0000 UTC m=+590.440598951 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls") pod "isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" (UID: "42416c69-5294-432a-b19b-c77544078842") : secret "isvc-raw-sklearn-batcher-b19ff-predictor-serving-cert" not found
Jul 28 17:26:53.007323 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.007258    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42416c69-5294-432a-b19b-c77544078842-kserve-provision-location\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.007494 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.007473    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-kube-rbac-proxy-sar-config\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.007624 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.007605    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-cabundle-cert\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.018364 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.018344    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-swrbf\" (UniqueName: \"kubernetes.io/projected/42416c69-5294-432a-b19b-c77544078842-kube-api-access-swrbf\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.510905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.510865    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.513133 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.513114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls\") pod \"isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") " pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.721399 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.721368    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:26:53.850961 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.850824    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"]
Jul 28 17:26:53.853974 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:26:53.853948    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod42416c69_5294_432a_b19b_c77544078842.slice/crio-e80fb19870e0ecabf0c7042fa472cc773a14d323c3f6a345a7b32449509e738d WatchSource:0}: Error finding container e80fb19870e0ecabf0c7042fa472cc773a14d323c3f6a345a7b32449509e738d: Status 404 returned error can't find the container with id e80fb19870e0ecabf0c7042fa472cc773a14d323c3f6a345a7b32449509e738d
Jul 28 17:26:53.988982 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.988947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerStarted","Data":"546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b"}
Jul 28 17:26:53.989100 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:53.989022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerStarted","Data":"e80fb19870e0ecabf0c7042fa472cc773a14d323c3f6a345a7b32449509e738d"}
Jul 28 17:26:58.007548 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:58.007463    2563 generic.go:356] "Generic (PLEG): container finished" podID="42416c69-5294-432a-b19b-c77544078842" containerID="546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b" exitCode=0
Jul 28 17:26:58.007996 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:26:58.007542    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerDied","Data":"546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b"}
Jul 28 17:27:11.073857 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:11.073822    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerStarted","Data":"7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408"}
Jul 28 17:27:13.083411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:13.083383    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerStarted","Data":"b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592"}
Jul 28 17:27:16.097441 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:16.097391    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerStarted","Data":"8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9"}
Jul 28 17:27:16.097834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:16.097526    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:27:16.124064 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:16.124020    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podStartSLOduration=3.394850307 podStartE2EDuration="24.124007428s" totalImagesPullingTime="16.988879211s" totalInitContainerRuntime="3.74027791s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:26:52 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:27:16.122497561 +0000 UTC m=+613.055896978" watchObservedRunningTime="2026-07-28 17:27:16.124007428 +0000 UTC m=+613.057406845"
Jul 28 17:27:17.102556 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:17.102524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:27:17.102978 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:17.102597    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:27:17.103831 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:17.103805    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:27:17.104441 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:17.104419    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:18.106583 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:18.106537    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:27:18.106987 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:18.106910    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:18.110273 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:18.110254    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:27:19.111482 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:19.111443    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:27:19.111898 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:19.111814    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:29.112235 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:29.112135    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:27:29.112692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:29.112651    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:39.111706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:39.111647    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:27:39.112243 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:39.112217    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:49.111514 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:49.111467    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:27:49.125525 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:49.111992    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:27:59.111563 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:59.111518    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:27:59.112031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:27:59.112009    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:09.112157 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:09.112115    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:28:09.112593 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:09.112489    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:19.111875 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:19.111823    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:28:19.112372 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:19.112337    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:29.112715 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:29.112666    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:28:29.113170 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:29.112801    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:28:37.961851 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:37.961811    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"]
Jul 28 17:28:37.962300 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:37.962114    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" containerID="cri-o://7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408" gracePeriod=30
Jul 28 17:28:37.962300 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:37.962138    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" containerID="cri-o://8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9" gracePeriod=30
Jul 28 17:28:37.962300 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:37.962155    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy" containerID="cri-o://b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592" gracePeriod=30
Jul 28 17:28:38.003192 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.003163    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"]
Jul 28 17:28:38.009610 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.008629    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.011882 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.011860    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-04101-predictor-serving-cert\""
Jul 28 17:28:38.012005 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.011989    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-04101-kube-rbac-proxy-sar-config\""
Jul 28 17:28:38.022597 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.022577    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"]
Jul 28 17:28:38.085534 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.085508    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"]
Jul 28 17:28:38.088843 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.088828    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.091469 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.091441    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-04101-kube-rbac-proxy-sar-config\""
Jul 28 17:28:38.091601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.091585    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-04101-predictor-serving-cert\""
Jul 28 17:28:38.103514 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.103488    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"]
Jul 28 17:28:38.107035 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.107007    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 28 17:28:38.108330 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.108310    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.108398 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.108351    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.108398 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.108374    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2753f307-d5d0-44b3-8329-33afdf309c1f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.108557 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.108467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2753f307-d5d0-44b3-8329-33afdf309c1f-proxy-tls\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.108611 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.108573    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j56j8\" (UniqueName: \"kubernetes.io/projected/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-api-access-j56j8\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.209970 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.209943    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2753f307-d5d0-44b3-8329-33afdf309c1f-proxy-tls\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.209970 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.209972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca93f63-307f-46b9-a9b5-74e139aca176-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.210135 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j56j8\" (UniqueName: \"kubernetes.io/projected/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-api-access-j56j8\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.210135 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210035    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksds8\" (UniqueName: \"kubernetes.io/projected/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-api-access-ksds8\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.210135 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.210135 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.210135 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.210135 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210121    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-cabundle-cert\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.210455 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.210455 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2753f307-d5d0-44b3-8329-33afdf309c1f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.210653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2753f307-d5d0-44b3-8329-33afdf309c1f-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.210769 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210752    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.210769 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.210763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-cabundle-cert\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.212412 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.212366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2753f307-d5d0-44b3-8329-33afdf309c1f-proxy-tls\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.218508 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.218488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j56j8\" (UniqueName: \"kubernetes.io/projected/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-api-access-j56j8\") pod \"isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.311652 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.311629    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksds8\" (UniqueName: \"kubernetes.io/projected/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-api-access-ksds8\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.311792 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.311660    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.311792 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.311706    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.311792 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:28:38.311784    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-serving-cert: secret "isvc-xgboost-graph-raw-04101-predictor-serving-cert" not found
Jul 28 17:28:38.311952 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:28:38.311837    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls podName:6ca93f63-307f-46b9-a9b5-74e139aca176 nodeName:}" failed. No retries permitted until 2026-07-28 17:28:38.811821707 +0000 UTC m=+695.745221102 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls") pod "isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" (UID: "6ca93f63-307f-46b9-a9b5-74e139aca176") : secret "isvc-xgboost-graph-raw-04101-predictor-serving-cert" not found
Jul 28 17:28:38.311952 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.311885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-cabundle-cert\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.311952 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.311948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca93f63-307f-46b9-a9b5-74e139aca176-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.312241 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.312223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca93f63-307f-46b9-a9b5-74e139aca176-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.312301 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.312285    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.312371 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.312354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-cabundle-cert\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.319405 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.319385    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:38.320989 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.320972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksds8\" (UniqueName: \"kubernetes.io/projected/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-api-access-ksds8\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.417070 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.417042    2563 generic.go:356] "Generic (PLEG): container finished" podID="42416c69-5294-432a-b19b-c77544078842" containerID="b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592" exitCode=2
Jul 28 17:28:38.417199 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.417083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerDied","Data":"b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592"}
Jul 28 17:28:38.452134 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.452110    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"]
Jul 28 17:28:38.454693 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:28:38.454652    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2753f307_d5d0_44b3_8329_33afdf309c1f.slice/crio-348aebc3459025b1da250bd6847cd154b4c1052120e60505a2f0d012310a9fe8 WatchSource:0}: Error finding container 348aebc3459025b1da250bd6847cd154b4c1052120e60505a2f0d012310a9fe8: Status 404 returned error can't find the container with id 348aebc3459025b1da250bd6847cd154b4c1052120e60505a2f0d012310a9fe8
Jul 28 17:28:38.815940 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.815854    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.818200 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.818173    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls\") pod \"isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:38.998396 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:38.998352    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:28:39.112313 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:39.112269    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:28:39.112625 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:39.112585    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:39.129728 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:39.129708    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"]
Jul 28 17:28:39.131644 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:28:39.131616    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ca93f63_307f_46b9_a9b5_74e139aca176.slice/crio-2e8f4cf7953908c85a9ea28fa6cbf574f97c55773d915aed8cf265b41934f560 WatchSource:0}: Error finding container 2e8f4cf7953908c85a9ea28fa6cbf574f97c55773d915aed8cf265b41934f560: Status 404 returned error can't find the container with id 2e8f4cf7953908c85a9ea28fa6cbf574f97c55773d915aed8cf265b41934f560
Jul 28 17:28:39.422492 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:39.422396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" event={"ID":"6ca93f63-307f-46b9-a9b5-74e139aca176","Type":"ContainerStarted","Data":"b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18"}
Jul 28 17:28:39.422656 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:39.422469    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" event={"ID":"6ca93f63-307f-46b9-a9b5-74e139aca176","Type":"ContainerStarted","Data":"2e8f4cf7953908c85a9ea28fa6cbf574f97c55773d915aed8cf265b41934f560"}
Jul 28 17:28:39.423942 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:39.423914    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" event={"ID":"2753f307-d5d0-44b3-8329-33afdf309c1f","Type":"ContainerStarted","Data":"9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f"}
Jul 28 17:28:39.424041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:39.423978    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" event={"ID":"2753f307-d5d0-44b3-8329-33afdf309c1f","Type":"ContainerStarted","Data":"348aebc3459025b1da250bd6847cd154b4c1052120e60505a2f0d012310a9fe8"}
Jul 28 17:28:42.438429 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:42.438398    2563 generic.go:356] "Generic (PLEG): container finished" podID="42416c69-5294-432a-b19b-c77544078842" containerID="7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408" exitCode=0
Jul 28 17:28:42.438807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:42.438467    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerDied","Data":"7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408"}
Jul 28 17:28:42.439860 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:42.439837    2563 generic.go:356] "Generic (PLEG): container finished" podID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerID="9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f" exitCode=0
Jul 28 17:28:42.439935 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:42.439910    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" event={"ID":"2753f307-d5d0-44b3-8329-33afdf309c1f","Type":"ContainerDied","Data":"9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f"}
Jul 28 17:28:43.107380 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.107296    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 28 17:28:43.445322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.445247    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerID="b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18" exitCode=0
Jul 28 17:28:43.445793 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.445317    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" event={"ID":"6ca93f63-307f-46b9-a9b5-74e139aca176","Type":"ContainerDied","Data":"b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18"}
Jul 28 17:28:43.446625 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.446608    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:28:43.447504 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.447481    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" event={"ID":"2753f307-d5d0-44b3-8329-33afdf309c1f","Type":"ContainerStarted","Data":"7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c"}
Jul 28 17:28:43.447596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.447543    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" event={"ID":"2753f307-d5d0-44b3-8329-33afdf309c1f","Type":"ContainerStarted","Data":"f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922"}
Jul 28 17:28:43.447821 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.447802    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:43.447867 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.447858    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:43.449022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.448996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:28:43.501867 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:43.501814    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podStartSLOduration=2.687884876 podStartE2EDuration="6.501794491s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.813909615s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:28:43.499104101 +0000 UTC m=+700.432503521" watchObservedRunningTime="2026-07-28 17:28:43.501794491 +0000 UTC m=+700.435193909"
Jul 28 17:28:44.453785 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:44.453576    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:28:48.107326 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:48.107280    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 28 17:28:48.107807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:48.107423    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:28:49.112383 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:49.112338    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:28:49.112869 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:49.112781    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:49.458572 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:49.458497    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:28:49.459111 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:49.459082    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:28:53.107647 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:53.107604    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 28 17:28:58.107089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:58.107000    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 28 17:28:59.111452 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:59.111402    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.22:8080: connect: connection refused"
Jul 28 17:28:59.111948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:59.111560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:28:59.111948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:59.111808    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:28:59.111948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:59.111927    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:28:59.459856 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:28:59.459779    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:29:02.543514 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:02.543472    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" event={"ID":"6ca93f63-307f-46b9-a9b5-74e139aca176","Type":"ContainerStarted","Data":"547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9"}
Jul 28 17:29:02.543948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:02.543534    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" event={"ID":"6ca93f63-307f-46b9-a9b5-74e139aca176","Type":"ContainerStarted","Data":"1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8"}
Jul 28 17:29:02.543948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:02.543812    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:29:02.543948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:02.543861    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:29:02.545106 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:02.545077    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:29:02.566992 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:02.566947    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podStartSLOduration=2.766939284 podStartE2EDuration="24.566933355s" totalImagesPullingTime="18.143029286s" totalInitContainerRuntime="3.656964785s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:28:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:29:02.566068449 +0000 UTC m=+719.499467869" watchObservedRunningTime="2026-07-28 17:29:02.566933355 +0000 UTC m=+719.500332773"
Jul 28 17:29:03.106686 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:03.106640    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": dial tcp 10.133.0.22:8643: connect: connection refused"
Jul 28 17:29:03.547796 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:03.547706    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:29:08.116359 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.116332    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:29:08.182734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.182706    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-swrbf\" (UniqueName: \"kubernetes.io/projected/42416c69-5294-432a-b19b-c77544078842-kube-api-access-swrbf\") pod \"42416c69-5294-432a-b19b-c77544078842\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") "
Jul 28 17:29:08.182891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.182763    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42416c69-5294-432a-b19b-c77544078842-kserve-provision-location\") pod \"42416c69-5294-432a-b19b-c77544078842\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") "
Jul 28 17:29:08.182891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.182793    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-cabundle-cert\") pod \"42416c69-5294-432a-b19b-c77544078842\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") "
Jul 28 17:29:08.183007 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.182897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-kube-rbac-proxy-sar-config\") pod \"42416c69-5294-432a-b19b-c77544078842\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") "
Jul 28 17:29:08.183007 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.182963    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls\") pod \"42416c69-5294-432a-b19b-c77544078842\" (UID: \"42416c69-5294-432a-b19b-c77544078842\") "
Jul 28 17:29:08.183109 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.183065    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/42416c69-5294-432a-b19b-c77544078842-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "42416c69-5294-432a-b19b-c77544078842" (UID: "42416c69-5294-432a-b19b-c77544078842"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:29:08.183109 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.183099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "42416c69-5294-432a-b19b-c77544078842" (UID: "42416c69-5294-432a-b19b-c77544078842"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:29:08.183253 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.183235    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "42416c69-5294-432a-b19b-c77544078842" (UID: "42416c69-5294-432a-b19b-c77544078842"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:29:08.183317 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.183238    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/42416c69-5294-432a-b19b-c77544078842-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:29:08.183317 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.183276    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:29:08.184952 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.184932    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "42416c69-5294-432a-b19b-c77544078842" (UID: "42416c69-5294-432a-b19b-c77544078842"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:29:08.185033 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.185016    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42416c69-5294-432a-b19b-c77544078842-kube-api-access-swrbf" (OuterVolumeSpecName: "kube-api-access-swrbf") pod "42416c69-5294-432a-b19b-c77544078842" (UID: "42416c69-5294-432a-b19b-c77544078842"). InnerVolumeSpecName "kube-api-access-swrbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:29:08.284140 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.284064    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-swrbf\" (UniqueName: \"kubernetes.io/projected/42416c69-5294-432a-b19b-c77544078842-kube-api-access-swrbf\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:29:08.284140 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.284098    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/42416c69-5294-432a-b19b-c77544078842-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:29:08.284140 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.284113    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/42416c69-5294-432a-b19b-c77544078842-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:29:08.552573 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.552494    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:29:08.553003 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.552977    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:29:08.570158 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.570123    2563 generic.go:356] "Generic (PLEG): container finished" podID="42416c69-5294-432a-b19b-c77544078842" containerID="8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9" exitCode=0
Jul 28 17:29:08.570297 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.570203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"
Jul 28 17:29:08.570297 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.570208    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerDied","Data":"8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9"}
Jul 28 17:29:08.570297 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.570275    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" event={"ID":"42416c69-5294-432a-b19b-c77544078842","Type":"ContainerDied","Data":"e80fb19870e0ecabf0c7042fa472cc773a14d323c3f6a345a7b32449509e738d"}
Jul 28 17:29:08.570484 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.570307    2563 scope.go:117] "RemoveContainer" containerID="8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9"
Jul 28 17:29:08.578903 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.578886    2563 scope.go:117] "RemoveContainer" containerID="b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592"
Jul 28 17:29:08.586485 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.586468    2563 scope.go:117] "RemoveContainer" containerID="7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408"
Jul 28 17:29:08.593222 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.593207    2563 scope.go:117] "RemoveContainer" containerID="546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b"
Jul 28 17:29:08.596180 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.596160    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"]
Jul 28 17:29:08.600517 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.600496    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2"]
Jul 28 17:29:08.601370 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.601351    2563 scope.go:117] "RemoveContainer" containerID="8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9"
Jul 28 17:29:08.601705 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:29:08.601684    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9\": container with ID starting with 8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9 not found: ID does not exist" containerID="8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9"
Jul 28 17:29:08.601760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.601717    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9"} err="failed to get container status \"8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9\": rpc error: code = NotFound desc = could not find container \"8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9\": container with ID starting with 8c87836ee960682d40a47ef1e5d0d23a1095b806230b160fbfe596482a95fdd9 not found: ID does not exist"
Jul 28 17:29:08.601760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.601739    2563 scope.go:117] "RemoveContainer" containerID="b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592"
Jul 28 17:29:08.602004 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:29:08.601986    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592\": container with ID starting with b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592 not found: ID does not exist" containerID="b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592"
Jul 28 17:29:08.602066 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.602010    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592"} err="failed to get container status \"b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592\": rpc error: code = NotFound desc = could not find container \"b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592\": container with ID starting with b1216d9c46f15fc08a18f022ae8e896eadc1e36eee171733019e85a664544592 not found: ID does not exist"
Jul 28 17:29:08.602066 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.602026    2563 scope.go:117] "RemoveContainer" containerID="7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408"
Jul 28 17:29:08.602297 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:29:08.602273    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408\": container with ID starting with 7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408 not found: ID does not exist" containerID="7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408"
Jul 28 17:29:08.602353 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.602304    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408"} err="failed to get container status \"7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408\": rpc error: code = NotFound desc = could not find container \"7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408\": container with ID starting with 7d0ec149e186d8bdc7707474c6f4e41c573409ad8174cf773077850748c24408 not found: ID does not exist"
Jul 28 17:29:08.602353 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.602324    2563 scope.go:117] "RemoveContainer" containerID="546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b"
Jul 28 17:29:08.602595 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:29:08.602573    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b\": container with ID starting with 546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b not found: ID does not exist" containerID="546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b"
Jul 28 17:29:08.602640 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:08.602601    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b"} err="failed to get container status \"546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b\": rpc error: code = NotFound desc = could not find container \"546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b\": container with ID starting with 546da1c198a7a0691c9fa20f8b026c3505c5120c3146f608566d7acc247ba52b not found: ID does not exist"
Jul 28 17:29:09.107190 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:09.107150    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-raw-sklearn-batcher-b19ff-predictor-6c4fd464-lssh2" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.22:8643/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 28 17:29:09.459145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:09.459064    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:29:09.596095 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:09.596062    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="42416c69-5294-432a-b19b-c77544078842" path="/var/lib/kubelet/pods/42416c69-5294-432a-b19b-c77544078842/volumes"
Jul 28 17:29:18.553189 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:18.553151    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:29:19.459690 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:19.459641    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:29:28.553821 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:28.553783    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:29:29.459325 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:29.459290    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:29:38.553909 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:38.553862    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:29:39.459226 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:39.459184    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:29:48.553425 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:48.553388    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:29:49.459220 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:49.459182    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:29:58.553817 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:58.553784    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:29:59.459842 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:29:59.459809    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:30:28.208622 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.208551    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"]
Jul 28 17:30:28.209062 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.208843    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" containerID="cri-o://f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922" gracePeriod=30
Jul 28 17:30:28.209062 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.208864    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kube-rbac-proxy" containerID="cri-o://7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c" gracePeriod=30
Jul 28 17:30:28.257198 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257175    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"]
Jul 28 17:30:28.257540 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257526    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container"
Jul 28 17:30:28.257588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257542    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container"
Jul 28 17:30:28.257588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257555    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent"
Jul 28 17:30:28.257588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257560    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent"
Jul 28 17:30:28.257588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257569    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="storage-initializer"
Jul 28 17:30:28.257588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257574    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="storage-initializer"
Jul 28 17:30:28.257588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257587    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy"
Jul 28 17:30:28.257851 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257593    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy"
Jul 28 17:30:28.257851 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257658    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="agent"
Jul 28 17:30:28.257851 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257686    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kube-rbac-proxy"
Jul 28 17:30:28.257851 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.257698    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="42416c69-5294-432a-b19b-c77544078842" containerName="kserve-container"
Jul 28 17:30:28.260596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.260576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.263092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.263066    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-83c93-kube-rbac-proxy-sar-config\""
Jul 28 17:30:28.263193 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.263124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-graph-raw-hpa-83c93-predictor-serving-cert\""
Jul 28 17:30:28.274101 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.274058    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"]
Jul 28 17:30:28.323866 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.323839    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.323994 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.323890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znmfw\" (UniqueName: \"kubernetes.io/projected/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-api-access-znmfw\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.323994 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.323960    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.324065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.324021    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.324099 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.324067    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.340916 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.340893    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"]
Jul 28 17:30:28.343380 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.343366    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.345885 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.345860    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-83c93-predictor-serving-cert\""
Jul 28 17:30:28.345992 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.345965    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-graph-raw-hpa-83c93-kube-rbac-proxy-sar-config\""
Jul 28 17:30:28.356792 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.356772    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"]
Jul 28 17:30:28.424522 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.424645 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424556    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69ee67c2-cf04-4f37-9733-5c31bf8fe986-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.424645 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424588    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znmfw\" (UniqueName: \"kubernetes.io/projected/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-api-access-znmfw\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.424645 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.424791 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424713    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvswt\" (UniqueName: \"kubernetes.io/projected/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-api-access-tvswt\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.424791 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424758    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.424864 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.424864 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424844    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.424937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424873    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.424937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.424900    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.425029 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.425009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kserve-provision-location\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.425310 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.425263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.425634 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.425612    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-cabundle-cert\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.427661 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.427639    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-proxy-tls\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.434223 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.434198    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-znmfw\" (UniqueName: \"kubernetes.io/projected/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-api-access-znmfw\") pod \"isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") " pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.446584 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.446563    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"]
Jul 28 17:30:28.446817 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.446788    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" containerID="cri-o://1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8" gracePeriod=30
Jul 28 17:30:28.446919 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.446823    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kube-rbac-proxy" containerID="cri-o://547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9" gracePeriod=30
Jul 28 17:30:28.525881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.525821    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.525881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.525851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.526039 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.525897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69ee67c2-cf04-4f37-9733-5c31bf8fe986-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.526039 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.525934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvswt\" (UniqueName: \"kubernetes.io/projected/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-api-access-tvswt\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.526039 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.525952    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.526252 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.526227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kserve-provision-location\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.526569 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.526549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-cabundle-cert\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.526637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.526569    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.528237 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.528212    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69ee67c2-cf04-4f37-9733-5c31bf8fe986-proxy-tls\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.534518 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.534500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvswt\" (UniqueName: \"kubernetes.io/projected/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-api-access-tvswt\") pod \"isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") " pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.548504 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.548478    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.24:8643/healthz\": dial tcp 10.133.0.24:8643: connect: connection refused"
Jul 28 17:30:28.553067 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.553034    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.24:8080: connect: connection refused"
Jul 28 17:30:28.571538 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.571517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:28.652763 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.652721    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:28.702313 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.699224    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"]
Jul 28 17:30:28.710448 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:30:28.710408    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eec4de5_ba0f_4579_84a1_7b8c01bb1e7d.slice/crio-ca8c567f90eda244b7a042afdf22d730b955795edbefe41f935187c318d4e64f WatchSource:0}: Error finding container ca8c567f90eda244b7a042afdf22d730b955795edbefe41f935187c318d4e64f: Status 404 returned error can't find the container with id ca8c567f90eda244b7a042afdf22d730b955795edbefe41f935187c318d4e64f
Jul 28 17:30:28.787298 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.787275    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"]
Jul 28 17:30:28.788879 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:30:28.788852    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69ee67c2_cf04_4f37_9733_5c31bf8fe986.slice/crio-fd7799323c3b19671f8a191b22252bc0f13493c9409d3d80a25050de1f00ca36 WatchSource:0}: Error finding container fd7799323c3b19671f8a191b22252bc0f13493c9409d3d80a25050de1f00ca36: Status 404 returned error can't find the container with id fd7799323c3b19671f8a191b22252bc0f13493c9409d3d80a25050de1f00ca36
Jul 28 17:30:28.893307 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.893278    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerID="547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9" exitCode=2
Jul 28 17:30:28.893432 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.893360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" event={"ID":"6ca93f63-307f-46b9-a9b5-74e139aca176","Type":"ContainerDied","Data":"547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9"}
Jul 28 17:30:28.895146 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.895122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" event={"ID":"69ee67c2-cf04-4f37-9733-5c31bf8fe986","Type":"ContainerStarted","Data":"da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65"}
Jul 28 17:30:28.895336 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.895318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" event={"ID":"69ee67c2-cf04-4f37-9733-5c31bf8fe986","Type":"ContainerStarted","Data":"fd7799323c3b19671f8a191b22252bc0f13493c9409d3d80a25050de1f00ca36"}
Jul 28 17:30:28.897715 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.897682    2563 generic.go:356] "Generic (PLEG): container finished" podID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerID="7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c" exitCode=2
Jul 28 17:30:28.897817 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.897698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" event={"ID":"2753f307-d5d0-44b3-8329-33afdf309c1f","Type":"ContainerDied","Data":"7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c"}
Jul 28 17:30:28.899533 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.899508    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" event={"ID":"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d","Type":"ContainerStarted","Data":"e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1"}
Jul 28 17:30:28.899656 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:28.899575    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" event={"ID":"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d","Type":"ContainerStarted","Data":"ca8c567f90eda244b7a042afdf22d730b955795edbefe41f935187c318d4e64f"}
Jul 28 17:30:29.454205 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:29.454164    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.23:8643/healthz\": dial tcp 10.133.0.23:8643: connect: connection refused"
Jul 28 17:30:29.459595 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:29.459564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.23:8080: connect: connection refused"
Jul 28 17:30:31.884412 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.884392    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:30:31.917479 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.917454    2563 generic.go:356] "Generic (PLEG): container finished" podID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerID="1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8" exitCode=0
Jul 28 17:30:31.917686 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.917488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" event={"ID":"6ca93f63-307f-46b9-a9b5-74e139aca176","Type":"ContainerDied","Data":"1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8"}
Jul 28 17:30:31.917686 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.917519    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"
Jul 28 17:30:31.917686 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.917535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29" event={"ID":"6ca93f63-307f-46b9-a9b5-74e139aca176","Type":"ContainerDied","Data":"2e8f4cf7953908c85a9ea28fa6cbf574f97c55773d915aed8cf265b41934f560"}
Jul 28 17:30:31.917686 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.917571    2563 scope.go:117] "RemoveContainer" containerID="547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9"
Jul 28 17:30:31.927145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.927126    2563 scope.go:117] "RemoveContainer" containerID="1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8"
Jul 28 17:30:31.934168 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.934150    2563 scope.go:117] "RemoveContainer" containerID="b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18"
Jul 28 17:30:31.940775 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.940746    2563 scope.go:117] "RemoveContainer" containerID="547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9"
Jul 28 17:30:31.941002 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:30:31.940984    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9\": container with ID starting with 547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9 not found: ID does not exist" containerID="547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9"
Jul 28 17:30:31.941052 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.941010    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9"} err="failed to get container status \"547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9\": rpc error: code = NotFound desc = could not find container \"547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9\": container with ID starting with 547218d6059bf4d1c1d8bae393fc8092880ec4130040cca6e1f1124518d552d9 not found: ID does not exist"
Jul 28 17:30:31.941052 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.941031    2563 scope.go:117] "RemoveContainer" containerID="1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8"
Jul 28 17:30:31.941254 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:30:31.941235    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8\": container with ID starting with 1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8 not found: ID does not exist" containerID="1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8"
Jul 28 17:30:31.941293 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.941265    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8"} err="failed to get container status \"1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8\": rpc error: code = NotFound desc = could not find container \"1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8\": container with ID starting with 1ef163b148191cabb9fcdf197a0faca1eddc8efd80db959c67298fd8b3620eb8 not found: ID does not exist"
Jul 28 17:30:31.941293 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.941281    2563 scope.go:117] "RemoveContainer" containerID="b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18"
Jul 28 17:30:31.941535 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:30:31.941518    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18\": container with ID starting with b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18 not found: ID does not exist" containerID="b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18"
Jul 28 17:30:31.941582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.941539    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18"} err="failed to get container status \"b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18\": rpc error: code = NotFound desc = could not find container \"b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18\": container with ID starting with b1200d60ad42bce228c540dc1be51d37e562d68c72fce719d7b23a8fc25ceb18 not found: ID does not exist"
Jul 28 17:30:31.959847 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.959805    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ksds8\" (UniqueName: \"kubernetes.io/projected/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-api-access-ksds8\") pod \"6ca93f63-307f-46b9-a9b5-74e139aca176\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") "
Jul 28 17:30:31.959847 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.959834    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca93f63-307f-46b9-a9b5-74e139aca176-kserve-provision-location\") pod \"6ca93f63-307f-46b9-a9b5-74e139aca176\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") "
Jul 28 17:30:31.959957 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.959866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-rbac-proxy-sar-config\") pod \"6ca93f63-307f-46b9-a9b5-74e139aca176\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") "
Jul 28 17:30:31.959957 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.959882    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls\") pod \"6ca93f63-307f-46b9-a9b5-74e139aca176\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") "
Jul 28 17:30:31.959957 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.959917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-cabundle-cert\") pod \"6ca93f63-307f-46b9-a9b5-74e139aca176\" (UID: \"6ca93f63-307f-46b9-a9b5-74e139aca176\") "
Jul 28 17:30:31.960186 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.960163    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ca93f63-307f-46b9-a9b5-74e139aca176-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6ca93f63-307f-46b9-a9b5-74e139aca176" (UID: "6ca93f63-307f-46b9-a9b5-74e139aca176"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:31.960258 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.960176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ca93f63-307f-46b9-a9b5-74e139aca176" (UID: "6ca93f63-307f-46b9-a9b5-74e139aca176"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:31.960368 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.960348    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6ca93f63-307f-46b9-a9b5-74e139aca176" (UID: "6ca93f63-307f-46b9-a9b5-74e139aca176"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:31.961742 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.961721    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-api-access-ksds8" (OuterVolumeSpecName: "kube-api-access-ksds8") pod "6ca93f63-307f-46b9-a9b5-74e139aca176" (UID: "6ca93f63-307f-46b9-a9b5-74e139aca176"). InnerVolumeSpecName "kube-api-access-ksds8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:31.961925 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:31.961899    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ca93f63-307f-46b9-a9b5-74e139aca176" (UID: "6ca93f63-307f-46b9-a9b5-74e139aca176"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:32.061189 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.061160    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.061189 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.061184    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ca93f63-307f-46b9-a9b5-74e139aca176-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.061189 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.061195    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6ca93f63-307f-46b9-a9b5-74e139aca176-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.061370 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.061205    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksds8\" (UniqueName: \"kubernetes.io/projected/6ca93f63-307f-46b9-a9b5-74e139aca176-kube-api-access-ksds8\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.061370 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.061214    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6ca93f63-307f-46b9-a9b5-74e139aca176-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.241819 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.241790    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"]
Jul 28 17:30:32.245420 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.245398    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-04101-predictor-867b8d5d7c-djg29"]
Jul 28 17:30:32.543077 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.543054    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:30:32.666878 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.666848    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2753f307-d5d0-44b3-8329-33afdf309c1f-kserve-provision-location\") pod \"2753f307-d5d0-44b3-8329-33afdf309c1f\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") "
Jul 28 17:30:32.667019 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.666891    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-cabundle-cert\") pod \"2753f307-d5d0-44b3-8329-33afdf309c1f\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") "
Jul 28 17:30:32.667019 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.666989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2753f307-d5d0-44b3-8329-33afdf309c1f-proxy-tls\") pod \"2753f307-d5d0-44b3-8329-33afdf309c1f\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") "
Jul 28 17:30:32.667019 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.667015    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-rbac-proxy-sar-config\") pod \"2753f307-d5d0-44b3-8329-33afdf309c1f\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") "
Jul 28 17:30:32.667176 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.667040    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j56j8\" (UniqueName: \"kubernetes.io/projected/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-api-access-j56j8\") pod \"2753f307-d5d0-44b3-8329-33afdf309c1f\" (UID: \"2753f307-d5d0-44b3-8329-33afdf309c1f\") "
Jul 28 17:30:32.667228 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.667176    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2753f307-d5d0-44b3-8329-33afdf309c1f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2753f307-d5d0-44b3-8329-33afdf309c1f" (UID: "2753f307-d5d0-44b3-8329-33afdf309c1f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:30:32.667296 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.667277    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2753f307-d5d0-44b3-8329-33afdf309c1f-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.667361 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.667281    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2753f307-d5d0-44b3-8329-33afdf309c1f" (UID: "2753f307-d5d0-44b3-8329-33afdf309c1f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:32.667361 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.667322    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2753f307-d5d0-44b3-8329-33afdf309c1f" (UID: "2753f307-d5d0-44b3-8329-33afdf309c1f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:30:32.668993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.668973    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2753f307-d5d0-44b3-8329-33afdf309c1f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2753f307-d5d0-44b3-8329-33afdf309c1f" (UID: "2753f307-d5d0-44b3-8329-33afdf309c1f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:30:32.669055 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.669026    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-api-access-j56j8" (OuterVolumeSpecName: "kube-api-access-j56j8") pod "2753f307-d5d0-44b3-8329-33afdf309c1f" (UID: "2753f307-d5d0-44b3-8329-33afdf309c1f"). InnerVolumeSpecName "kube-api-access-j56j8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:30:32.768490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.768457    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2753f307-d5d0-44b3-8329-33afdf309c1f-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.768490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.768487    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.768490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.768496    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j56j8\" (UniqueName: \"kubernetes.io/projected/2753f307-d5d0-44b3-8329-33afdf309c1f-kube-api-access-j56j8\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.768644 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.768504    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2753f307-d5d0-44b3-8329-33afdf309c1f-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:30:32.923206 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.923180    2563 generic.go:356] "Generic (PLEG): container finished" podID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerID="da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65" exitCode=0
Jul 28 17:30:32.923579 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.923243    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" event={"ID":"69ee67c2-cf04-4f37-9733-5c31bf8fe986","Type":"ContainerDied","Data":"da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65"}
Jul 28 17:30:32.925166 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.925143    2563 generic.go:356] "Generic (PLEG): container finished" podID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerID="f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922" exitCode=0
Jul 28 17:30:32.925242 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.925176    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" event={"ID":"2753f307-d5d0-44b3-8329-33afdf309c1f","Type":"ContainerDied","Data":"f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922"}
Jul 28 17:30:32.925242 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.925199    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"
Jul 28 17:30:32.925242 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.925219    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t" event={"ID":"2753f307-d5d0-44b3-8329-33afdf309c1f","Type":"ContainerDied","Data":"348aebc3459025b1da250bd6847cd154b4c1052120e60505a2f0d012310a9fe8"}
Jul 28 17:30:32.925379 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.925275    2563 scope.go:117] "RemoveContainer" containerID="7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c"
Jul 28 17:30:32.926949 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.926931    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerID="e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1" exitCode=0
Jul 28 17:30:32.927015 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.926955    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" event={"ID":"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d","Type":"ContainerDied","Data":"e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1"}
Jul 28 17:30:32.936637 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.936621    2563 scope.go:117] "RemoveContainer" containerID="f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922"
Jul 28 17:30:32.949291 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.949260    2563 scope.go:117] "RemoveContainer" containerID="9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f"
Jul 28 17:30:32.958131 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.958111    2563 scope.go:117] "RemoveContainer" containerID="7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c"
Jul 28 17:30:32.958374 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:30:32.958357    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c\": container with ID starting with 7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c not found: ID does not exist" containerID="7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c"
Jul 28 17:30:32.958419 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.958380    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c"} err="failed to get container status \"7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c\": rpc error: code = NotFound desc = could not find container \"7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c\": container with ID starting with 7035cd17819f7694aa1b867911119d6c18de64ff00557d9c66fd23aac62cf47c not found: ID does not exist"
Jul 28 17:30:32.958419 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.958396    2563 scope.go:117] "RemoveContainer" containerID="f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922"
Jul 28 17:30:32.958652 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:30:32.958633    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922\": container with ID starting with f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922 not found: ID does not exist" containerID="f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922"
Jul 28 17:30:32.958741 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.958657    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922"} err="failed to get container status \"f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922\": rpc error: code = NotFound desc = could not find container \"f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922\": container with ID starting with f06405c5b1926d47e78df6c964ab7da587b375ff12651e6ec7b32e7bfbcf6922 not found: ID does not exist"
Jul 28 17:30:32.958741 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.958688    2563 scope.go:117] "RemoveContainer" containerID="9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f"
Jul 28 17:30:32.958909 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:30:32.958896    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f\": container with ID starting with 9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f not found: ID does not exist" containerID="9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f"
Jul 28 17:30:32.958949 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.958916    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f"} err="failed to get container status \"9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f\": rpc error: code = NotFound desc = could not find container \"9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f\": container with ID starting with 9995f53e2dbbf34c09ed074e7e404ec5698b0515bc20af80f1ec19ce1cbc0d8f not found: ID does not exist"
Jul 28 17:30:32.978206 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.978183    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"]
Jul 28 17:30:32.982664 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:32.982642    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-04101-predictor-54f484c8d4-kzg8t"]
Jul 28 17:30:33.596943 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.596910    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" path="/var/lib/kubelet/pods/2753f307-d5d0-44b3-8329-33afdf309c1f/volumes"
Jul 28 17:30:33.597401 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.597389    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" path="/var/lib/kubelet/pods/6ca93f63-307f-46b9-a9b5-74e139aca176/volumes"
Jul 28 17:30:33.933471 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.933398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" event={"ID":"69ee67c2-cf04-4f37-9733-5c31bf8fe986","Type":"ContainerStarted","Data":"eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80"}
Jul 28 17:30:33.933471 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.933464    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" event={"ID":"69ee67c2-cf04-4f37-9733-5c31bf8fe986","Type":"ContainerStarted","Data":"e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61"}
Jul 28 17:30:33.933969 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.933707    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:33.933969 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.933884    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:33.935440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.935411    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:30:33.936527 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.936507    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" event={"ID":"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d","Type":"ContainerStarted","Data":"711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2"}
Jul 28 17:30:33.936609 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.936550    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" event={"ID":"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d","Type":"ContainerStarted","Data":"4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762"}
Jul 28 17:30:33.936783 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.936766    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:33.936830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.936806    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:33.937899 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.937862    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:30:33.957109 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.957070    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podStartSLOduration=2.345782131 podStartE2EDuration="5.957057006s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.611274875s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:33.954928066 +0000 UTC m=+810.888327509" watchObservedRunningTime="2026-07-28 17:30:33.957057006 +0000 UTC m=+810.890456423"
Jul 28 17:30:33.976489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:33.976449    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podStartSLOduration=2.416039181 podStartE2EDuration="5.976437345s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.560398164s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:30:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:30:33.974876604 +0000 UTC m=+810.908276023" watchObservedRunningTime="2026-07-28 17:30:33.976437345 +0000 UTC m=+810.909836762"
Jul 28 17:30:34.941005 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:34.940961    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:30:34.941479 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:34.941133    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:30:39.945850 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:39.945820    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:30:39.946254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:39.946209    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:30:39.946320 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:39.946295    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:30:39.946658 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:39.946638    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:30:49.946531 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:49.946488    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:30:49.946966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:49.946549    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:30:59.947170 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:59.947129    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:30:59.947581 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:30:59.947129    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:31:09.946567 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:09.946526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:31:09.947076 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:09.946667    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:31:19.946484 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:19.946433    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:31:19.946912 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:19.946572    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:31:29.946657 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:29.946616    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:31:29.947166 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:29.946821    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:31:39.947279 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:39.947238    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:31:42.591911 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:42.591864    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:31:52.592484 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:31:52.592454    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:32:18.505645 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.505613    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"]
Jul 28 17:32:18.506183 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.505877    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" containerID="cri-o://4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762" gracePeriod=30
Jul 28 17:32:18.506183 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.505938    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kube-rbac-proxy" containerID="cri-o://711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2" gracePeriod=30
Jul 28 17:32:18.538004 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.537976    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"]
Jul 28 17:32:18.538299 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538287    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container"
Jul 28 17:32:18.538299 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538300    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538310    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538316    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538325    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kube-rbac-proxy"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538331    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kube-rbac-proxy"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538342    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="storage-initializer"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538347    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="storage-initializer"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538356    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kube-rbac-proxy"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538361    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kube-rbac-proxy"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538376    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="storage-initializer"
Jul 28 17:32:18.538382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538381    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="storage-initializer"
Jul 28 17:32:18.538662 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538432    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kserve-container"
Jul 28 17:32:18.538662 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538444    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kserve-container"
Jul 28 17:32:18.538662 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538452    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="2753f307-d5d0-44b3-8329-33afdf309c1f" containerName="kube-rbac-proxy"
Jul 28 17:32:18.538662 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.538458    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ca93f63-307f-46b9-a9b5-74e139aca176" containerName="kube-rbac-proxy"
Jul 28 17:32:18.541719 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.541704    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:18.544224 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.544209    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-d84a6-kube-rbac-proxy-sar-config\""
Jul 28 17:32:18.544442 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.544424    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-raw-d84a6-predictor-serving-cert\""
Jul 28 17:32:18.550800 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.550774    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"]
Jul 28 17:32:18.609328 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.609304    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"]
Jul 28 17:32:18.609557 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.609533    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" containerID="cri-o://e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61" gracePeriod=30
Jul 28 17:32:18.609633 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.609552    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kube-rbac-proxy" containerID="cri-o://eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80" gracePeriod=30
Jul 28 17:32:18.691641 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.691612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:18.691786 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.691644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfzt\" (UniqueName: \"kubernetes.io/projected/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-api-access-9qfzt\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:18.691786 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.691666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:18.793074 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.793016    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfzt\" (UniqueName: \"kubernetes.io/projected/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-api-access-9qfzt\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:18.793074 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.793050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:18.793210 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.793123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:18.793263 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:18.793230    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-serving-cert: secret "message-dumper-raw-d84a6-predictor-serving-cert" not found
Jul 28 17:32:18.793326 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:18.793313    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls podName:d66c7a93-c12f-4185-8bf9-b5e836a448f6 nodeName:}" failed. No retries permitted until 2026-07-28 17:32:19.293292463 +0000 UTC m=+916.226691867 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls") pod "message-dumper-raw-d84a6-predictor-6478888887-8rwd8" (UID: "d66c7a93-c12f-4185-8bf9-b5e836a448f6") : secret "message-dumper-raw-d84a6-predictor-serving-cert" not found
Jul 28 17:32:18.793695 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.793661    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-rbac-proxy-sar-config\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:18.802437 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:18.802411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfzt\" (UniqueName: \"kubernetes.io/projected/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-api-access-9qfzt\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:19.296191 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.296163    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:19.298644 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.298608    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls\") pod \"message-dumper-raw-d84a6-predictor-6478888887-8rwd8\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") " pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:19.360808 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.360780    2563 generic.go:356] "Generic (PLEG): container finished" podID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerID="eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80" exitCode=2
Jul 28 17:32:19.360931 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.360852    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" event={"ID":"69ee67c2-cf04-4f37-9733-5c31bf8fe986","Type":"ContainerDied","Data":"eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80"}
Jul 28 17:32:19.362875 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.362851    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerID="711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2" exitCode=2
Jul 28 17:32:19.362986 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.362894    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" event={"ID":"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d","Type":"ContainerDied","Data":"711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2"}
Jul 28 17:32:19.452156 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.452122    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:19.577513 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.577487    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"]
Jul 28 17:32:19.578952 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:32:19.578927    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66c7a93_c12f_4185_8bf9_b5e836a448f6.slice/crio-74f67ee524630400cbe4cda8188b726777b417c727b322e97f5eec65e217a6bf WatchSource:0}: Error finding container 74f67ee524630400cbe4cda8188b726777b417c727b322e97f5eec65e217a6bf: Status 404 returned error can't find the container with id 74f67ee524630400cbe4cda8188b726777b417c727b322e97f5eec65e217a6bf
Jul 28 17:32:19.941320 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.941234    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.26:8643/healthz\": dial tcp 10.133.0.26:8643: connect: connection refused"
Jul 28 17:32:19.941463 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.941234    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.25:8643/healthz\": dial tcp 10.133.0.25:8643: connect: connection refused"
Jul 28 17:32:19.947091 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:19.947065    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.26:8080: connect: connection refused"
Jul 28 17:32:20.371654 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:20.371613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" event={"ID":"d66c7a93-c12f-4185-8bf9-b5e836a448f6","Type":"ContainerStarted","Data":"74f67ee524630400cbe4cda8188b726777b417c727b322e97f5eec65e217a6bf"}
Jul 28 17:32:21.377792 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:21.377755    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" event={"ID":"d66c7a93-c12f-4185-8bf9-b5e836a448f6","Type":"ContainerStarted","Data":"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7"}
Jul 28 17:32:21.378276 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:21.377819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" event={"ID":"d66c7a93-c12f-4185-8bf9-b5e836a448f6","Type":"ContainerStarted","Data":"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79"}
Jul 28 17:32:21.378276 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:21.377853    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:21.378276 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:21.377955    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:21.379494 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:21.379476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:21.398792 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:21.398739    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" podStartSLOduration=2.351894547 podStartE2EDuration="3.398725368s" totalImagesPullingTime="1.046830821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:21.396286487 +0000 UTC m=+918.329685905" watchObservedRunningTime="2026-07-28 17:32:21.398725368 +0000 UTC m=+918.332124785"
Jul 28 17:32:22.058038 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.058013    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:32:22.222556 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.222475    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tvswt\" (UniqueName: \"kubernetes.io/projected/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-api-access-tvswt\") pod \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") "
Jul 28 17:32:22.222556 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.222516    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69ee67c2-cf04-4f37-9733-5c31bf8fe986-proxy-tls\") pod \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") "
Jul 28 17:32:22.222556 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.222555    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-cabundle-cert\") pod \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") "
Jul 28 17:32:22.222850 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.222693    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kserve-provision-location\") pod \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") "
Jul 28 17:32:22.222850 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.222774    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-rbac-proxy-sar-config\") pod \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\" (UID: \"69ee67c2-cf04-4f37-9733-5c31bf8fe986\") "
Jul 28 17:32:22.222961 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.222885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "69ee67c2-cf04-4f37-9733-5c31bf8fe986" (UID: "69ee67c2-cf04-4f37-9733-5c31bf8fe986"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:22.223078 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.223052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "69ee67c2-cf04-4f37-9733-5c31bf8fe986" (UID: "69ee67c2-cf04-4f37-9733-5c31bf8fe986"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:22.223133 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.223108    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:22.223133 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.223115    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "69ee67c2-cf04-4f37-9733-5c31bf8fe986" (UID: "69ee67c2-cf04-4f37-9733-5c31bf8fe986"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:22.224735 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.224707    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69ee67c2-cf04-4f37-9733-5c31bf8fe986-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "69ee67c2-cf04-4f37-9733-5c31bf8fe986" (UID: "69ee67c2-cf04-4f37-9733-5c31bf8fe986"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:22.224841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.224759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-api-access-tvswt" (OuterVolumeSpecName: "kube-api-access-tvswt") pod "69ee67c2-cf04-4f37-9733-5c31bf8fe986" (UID: "69ee67c2-cf04-4f37-9733-5c31bf8fe986"). InnerVolumeSpecName "kube-api-access-tvswt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:22.323865 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.323842    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tvswt\" (UniqueName: \"kubernetes.io/projected/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-api-access-tvswt\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:22.323865 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.323861    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/69ee67c2-cf04-4f37-9733-5c31bf8fe986-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:22.324014 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.323871    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:22.324014 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.323881    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/69ee67c2-cf04-4f37-9733-5c31bf8fe986-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:22.383091 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.383062    2563 generic.go:356] "Generic (PLEG): container finished" podID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerID="e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61" exitCode=0
Jul 28 17:32:22.383456 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.383132    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"
Jul 28 17:32:22.383456 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.383152    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" event={"ID":"69ee67c2-cf04-4f37-9733-5c31bf8fe986","Type":"ContainerDied","Data":"e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61"}
Jul 28 17:32:22.383456 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.383215    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw" event={"ID":"69ee67c2-cf04-4f37-9733-5c31bf8fe986","Type":"ContainerDied","Data":"fd7799323c3b19671f8a191b22252bc0f13493c9409d3d80a25050de1f00ca36"}
Jul 28 17:32:22.383456 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.383244    2563 scope.go:117] "RemoveContainer" containerID="eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80"
Jul 28 17:32:22.391588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.391575    2563 scope.go:117] "RemoveContainer" containerID="e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61"
Jul 28 17:32:22.405752 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.405734    2563 scope.go:117] "RemoveContainer" containerID="da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65"
Jul 28 17:32:22.409077 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.409057    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"]
Jul 28 17:32:22.413122 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.413104    2563 scope.go:117] "RemoveContainer" containerID="eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80"
Jul 28 17:32:22.413363 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:22.413346    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80\": container with ID starting with eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80 not found: ID does not exist" containerID="eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80"
Jul 28 17:32:22.413435 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.413369    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80"} err="failed to get container status \"eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80\": rpc error: code = NotFound desc = could not find container \"eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80\": container with ID starting with eaa3501d6e60599cbbe8098ce5941e70f333c482b5f82080a9949170fcb3ef80 not found: ID does not exist"
Jul 28 17:32:22.413435 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.413385    2563 scope.go:117] "RemoveContainer" containerID="e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61"
Jul 28 17:32:22.413435 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.413417    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-graph-raw-hpa-83c93-predictor-5bc85d7f7c-2npvw"]
Jul 28 17:32:22.413623 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:22.413607    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61\": container with ID starting with e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61 not found: ID does not exist" containerID="e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61"
Jul 28 17:32:22.413661 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.413627    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61"} err="failed to get container status \"e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61\": rpc error: code = NotFound desc = could not find container \"e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61\": container with ID starting with e64bd174f10022002746800ea9fc9d72bbb8e09cbaa189fdf5bcc3e818dd2b61 not found: ID does not exist"
Jul 28 17:32:22.413661 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.413641    2563 scope.go:117] "RemoveContainer" containerID="da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65"
Jul 28 17:32:22.413881 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:22.413864    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65\": container with ID starting with da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65 not found: ID does not exist" containerID="da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65"
Jul 28 17:32:22.413926 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.413887    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65"} err="failed to get container status \"da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65\": rpc error: code = NotFound desc = could not find container \"da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65\": container with ID starting with da1dd149b15cdf50584bbb7aff064e7cf1a409105f7265989c510e1e85c58d65 not found: ID does not exist"
Jul 28 17:32:22.591988 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.591951    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.25:8080: connect: connection refused"
Jul 28 17:32:22.941618 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:22.941598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:32:23.029951 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.029928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-proxy-tls\") pod \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") "
Jul 28 17:32:23.030068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.029984    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kserve-provision-location\") pod \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") "
Jul 28 17:32:23.030068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.030005    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-znmfw\" (UniqueName: \"kubernetes.io/projected/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-api-access-znmfw\") pod \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") "
Jul 28 17:32:23.030068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.030024    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-rbac-proxy-sar-config\") pod \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") "
Jul 28 17:32:23.030068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.030067    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-cabundle-cert\") pod \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\" (UID: \"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d\") "
Jul 28 17:32:23.030349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.030318    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" (UID: "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:32:23.030448 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.030426    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" (UID: "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:23.030528 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.030504    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" (UID: "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:32:23.031939 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.031916    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" (UID: "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:32:23.032074 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.032053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-api-access-znmfw" (OuterVolumeSpecName: "kube-api-access-znmfw") pod "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" (UID: "6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d"). InnerVolumeSpecName "kube-api-access-znmfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:32:23.131009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.130944    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.131009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.130968    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.131009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.130981    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.131009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.130992    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znmfw\" (UniqueName: \"kubernetes.io/projected/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-api-access-znmfw\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.131009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.131005    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:32:23.388806 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.388738    2563 generic.go:356] "Generic (PLEG): container finished" podID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerID="4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762" exitCode=0
Jul 28 17:32:23.389185 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.388803    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"
Jul 28 17:32:23.389185 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.388824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" event={"ID":"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d","Type":"ContainerDied","Data":"4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762"}
Jul 28 17:32:23.389185 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.388911    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2" event={"ID":"6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d","Type":"ContainerDied","Data":"ca8c567f90eda244b7a042afdf22d730b955795edbefe41f935187c318d4e64f"}
Jul 28 17:32:23.389185 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.388945    2563 scope.go:117] "RemoveContainer" containerID="711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2"
Jul 28 17:32:23.397261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.397243    2563 scope.go:117] "RemoveContainer" containerID="4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762"
Jul 28 17:32:23.404283 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.404267    2563 scope.go:117] "RemoveContainer" containerID="e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1"
Jul 28 17:32:23.410548 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.410528    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"]
Jul 28 17:32:23.411146 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.411130    2563 scope.go:117] "RemoveContainer" containerID="711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2"
Jul 28 17:32:23.411376 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:23.411358    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2\": container with ID starting with 711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2 not found: ID does not exist" containerID="711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2"
Jul 28 17:32:23.411433 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.411381    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2"} err="failed to get container status \"711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2\": rpc error: code = NotFound desc = could not find container \"711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2\": container with ID starting with 711dea51bfa175b43f0277fa048212c35de3bf75b80a0cc348668da86f2ea2f2 not found: ID does not exist"
Jul 28 17:32:23.411433 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.411395    2563 scope.go:117] "RemoveContainer" containerID="4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762"
Jul 28 17:32:23.411619 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:23.411601    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762\": container with ID starting with 4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762 not found: ID does not exist" containerID="4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762"
Jul 28 17:32:23.411659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.411626    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762"} err="failed to get container status \"4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762\": rpc error: code = NotFound desc = could not find container \"4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762\": container with ID starting with 4ea9c248061b3fea4bcc4777fc854af1de7e4ecd57aba8b6952f4965860fa762 not found: ID does not exist"
Jul 28 17:32:23.411659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.411643    2563 scope.go:117] "RemoveContainer" containerID="e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1"
Jul 28 17:32:23.411863 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:23.411849    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1\": container with ID starting with e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1 not found: ID does not exist" containerID="e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1"
Jul 28 17:32:23.411903 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.411866    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1"} err="failed to get container status \"e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1\": rpc error: code = NotFound desc = could not find container \"e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1\": container with ID starting with e6fc612feecc8f41af8875cc0e19faffff7622dca2ed05c8c083850b7a081ac1 not found: ID does not exist"
Jul 28 17:32:23.414592 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.414573    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-graph-raw-hpa-83c93-predictor-59c7c9d88d-bs4n2"]
Jul 28 17:32:23.596721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.596688    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" path="/var/lib/kubelet/pods/69ee67c2-cf04-4f37-9733-5c31bf8fe986/volumes"
Jul 28 17:32:23.597174 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:23.597161    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" path="/var/lib/kubelet/pods/6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d/volumes"
Jul 28 17:32:28.393932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:28.393907    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:32:38.592750 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.592715    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"]
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593020    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="storage-initializer"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593047    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="storage-initializer"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593062    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="storage-initializer"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593067    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="storage-initializer"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593077    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kube-rbac-proxy"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593083    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kube-rbac-proxy"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593097    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593102    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593108    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kube-rbac-proxy"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593113    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kube-rbac-proxy"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593119    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container"
Jul 28 17:32:38.593117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593125    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container"
Jul 28 17:32:38.593490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593174    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kube-rbac-proxy"
Jul 28 17:32:38.593490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593182    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kube-rbac-proxy"
Jul 28 17:32:38.593490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593189    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="69ee67c2-cf04-4f37-9733-5c31bf8fe986" containerName="kserve-container"
Jul 28 17:32:38.593490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.593196    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6eec4de5-ba0f-4579-84a1-7b8c01bb1e7d" containerName="kserve-container"
Jul 28 17:32:38.596504 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.596487    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.599239 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.599215    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:32:38.599468 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.599451    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-d84a6-kube-rbac-proxy-sar-config\""
Jul 28 17:32:38.599546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.599473    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-logger-raw-d84a6-predictor-serving-cert\""
Jul 28 17:32:38.599605 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.599586    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:32:38.607534 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.607516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"]
Jul 28 17:32:38.644473 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.644447    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.644581 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.644507    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.644581 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.644550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-cabundle-cert\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.644684 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.644586    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201ee33a-6197-4d86-90b2-0407348a27a6-kserve-provision-location\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.644684 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.644607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm8v7\" (UniqueName: \"kubernetes.io/projected/201ee33a-6197-4d86-90b2-0407348a27a6-kube-api-access-qm8v7\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.745044 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.745008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.745199 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.745062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.745199 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.745092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-cabundle-cert\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.745317 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.745188    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201ee33a-6197-4d86-90b2-0407348a27a6-kserve-provision-location\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.745317 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:38.745233    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-serving-cert: secret "isvc-logger-raw-d84a6-predictor-serving-cert" not found
Jul 28 17:32:38.745317 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.745267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm8v7\" (UniqueName: \"kubernetes.io/projected/201ee33a-6197-4d86-90b2-0407348a27a6-kube-api-access-qm8v7\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.745317 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:32:38.745302    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls podName:201ee33a-6197-4d86-90b2-0407348a27a6 nodeName:}" failed. No retries permitted until 2026-07-28 17:32:39.245284833 +0000 UTC m=+936.178684231 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls") pod "isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" (UID: "201ee33a-6197-4d86-90b2-0407348a27a6") : secret "isvc-logger-raw-d84a6-predictor-serving-cert" not found
Jul 28 17:32:38.745541 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.745518    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201ee33a-6197-4d86-90b2-0407348a27a6-kserve-provision-location\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.745761 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.745742    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-cabundle-cert\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.745854 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.745808    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-kube-rbac-proxy-sar-config\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:38.754041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:38.754016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm8v7\" (UniqueName: \"kubernetes.io/projected/201ee33a-6197-4d86-90b2-0407348a27a6-kube-api-access-qm8v7\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:39.248422 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:39.248386    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:39.250667 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:39.250644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls\") pod \"isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") " pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:39.507659 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:39.507584    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:39.639409 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:39.639370    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"]
Jul 28 17:32:39.642202 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:32:39.642176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod201ee33a_6197_4d86_90b2_0407348a27a6.slice/crio-fc7b68c70cbe7a7e52920146b5265e6ca1a8820ae34b786fe546a7e7ac2e264d WatchSource:0}: Error finding container fc7b68c70cbe7a7e52920146b5265e6ca1a8820ae34b786fe546a7e7ac2e264d: Status 404 returned error can't find the container with id fc7b68c70cbe7a7e52920146b5265e6ca1a8820ae34b786fe546a7e7ac2e264d
Jul 28 17:32:40.459535 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:40.459495    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerStarted","Data":"bce05794d261383601a47675aa9ec3429af566eba71d3b6f84d0749abd6d4af0"}
Jul 28 17:32:40.459722 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:40.459577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerStarted","Data":"fc7b68c70cbe7a7e52920146b5265e6ca1a8820ae34b786fe546a7e7ac2e264d"}
Jul 28 17:32:43.473073 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:43.473044    2563 generic.go:356] "Generic (PLEG): container finished" podID="201ee33a-6197-4d86-90b2-0407348a27a6" containerID="bce05794d261383601a47675aa9ec3429af566eba71d3b6f84d0749abd6d4af0" exitCode=0
Jul 28 17:32:43.473393 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:43.473125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerDied","Data":"bce05794d261383601a47675aa9ec3429af566eba71d3b6f84d0749abd6d4af0"}
Jul 28 17:32:44.479589 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.479553    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerStarted","Data":"63051469b8cb2ed0338862044e807f17cdde8e17b42a7ea348cd36ad45ee8b94"}
Jul 28 17:32:44.480075 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.479631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerStarted","Data":"b7a4a72188a8030538d8a340420550778b0c49937a7410375c0f25febbedb4cb"}
Jul 28 17:32:44.480075 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.479694    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerStarted","Data":"cf7f96c52a843a7efbdc6f60ef6d253e60aefcdcd3f41a2b709b97be9596d916"}
Jul 28 17:32:44.480075 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.479929    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:44.480075 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.479995    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:44.480075 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.480036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:44.481319 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.481285    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:32:44.481946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.481923    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:44.502489 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:44.502446    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podStartSLOduration=2.916322176 podStartE2EDuration="6.502434001s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.586111825s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:32:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:32:44.500077574 +0000 UTC m=+941.433477021" watchObservedRunningTime="2026-07-28 17:32:44.502434001 +0000 UTC m=+941.435833468"
Jul 28 17:32:45.483348 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:45.483309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:32:45.483824 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:45.483696    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:32:50.486987 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:50.486957    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:32:50.489222 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:50.487547    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:32:50.489222 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:32:50.488025    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:00.488059 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:00.488020    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:33:00.488575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:00.488553    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:10.487781 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:10.487734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:33:10.488170 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:10.488099    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:20.487943 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:20.487896    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:33:20.488411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:20.488386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:30.487548 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:30.487446    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:33:30.488073 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:30.487885    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:40.487507 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:40.487454    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:33:40.488090 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:40.487933    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:33:50.487822 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:50.487776    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:33:50.488239 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:33:50.488189    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:00.488615 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:00.488576    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:34:00.489022 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:00.488842    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:34:13.608565 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.608524    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-raw-d84a6-predictor-6478888887-8rwd8_d66c7a93-c12f-4185-8bf9-b5e836a448f6/kserve-container/0.log"
Jul 28 17:34:13.837333 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.837299    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"]
Jul 28 17:34:13.837714 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.837657    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" containerID="cri-o://cf7f96c52a843a7efbdc6f60ef6d253e60aefcdcd3f41a2b709b97be9596d916" gracePeriod=30
Jul 28 17:34:13.837714 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.837666    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy" containerID="cri-o://b7a4a72188a8030538d8a340420550778b0c49937a7410375c0f25febbedb4cb" gracePeriod=30
Jul 28 17:34:13.837953 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.837720    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" containerID="cri-o://63051469b8cb2ed0338862044e807f17cdde8e17b42a7ea348cd36ad45ee8b94" gracePeriod=30
Jul 28 17:34:13.887175 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.887121    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"]
Jul 28 17:34:13.890553 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.890539    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:13.893282 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.893260    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dca0e-kube-rbac-proxy-sar-config\""
Jul 28 17:34:13.893415 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.893294    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-scale-raw-dca0e-predictor-serving-cert\""
Jul 28 17:34:13.900757 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.900737    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"]
Jul 28 17:34:13.962125 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.962101    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f457ff-5ca7-41e9-b1a8-b85449e64d91-proxy-tls\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:13.962264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.962143    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:13.962264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.962207    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:13.962264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.962248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5grm\" (UniqueName: \"kubernetes.io/projected/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-api-access-m5grm\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:13.962437 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.962373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:13.979110 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.979083    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"]
Jul 28 17:34:13.979310 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.979281    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerName="kserve-container" containerID="cri-o://8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79" gracePeriod=30
Jul 28 17:34:13.979412 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:13.979318    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerName="kube-rbac-proxy" containerID="cri-o://fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7" gracePeriod=30
Jul 28 17:34:14.062826 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.062795    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.062961 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.062879    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f457ff-5ca7-41e9-b1a8-b85449e64d91-proxy-tls\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.062961 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.062909    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.062961 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.062933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.063133 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.062963    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5grm\" (UniqueName: \"kubernetes.io/projected/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-api-access-m5grm\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.063276 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.063251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kserve-provision-location\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.063494 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.063471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.063578 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.063530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-cabundle-cert\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.065243 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.065222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f457ff-5ca7-41e9-b1a8-b85449e64d91-proxy-tls\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.072307 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.072283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5grm\" (UniqueName: \"kubernetes.io/projected/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-api-access-m5grm\") pod \"isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") " pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.200849 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.200811    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:14.225754 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.225732    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:34:14.264905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.264361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qfzt\" (UniqueName: \"kubernetes.io/projected/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-api-access-9qfzt\") pod \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") "
Jul 28 17:34:14.264905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.264405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls\") pod \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") "
Jul 28 17:34:14.264905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.264455    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-rbac-proxy-sar-config\") pod \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\" (UID: \"d66c7a93-c12f-4185-8bf9-b5e836a448f6\") "
Jul 28 17:34:14.265279 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.265256    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d66c7a93-c12f-4185-8bf9-b5e836a448f6" (UID: "d66c7a93-c12f-4185-8bf9-b5e836a448f6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:14.266848 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.266822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d66c7a93-c12f-4185-8bf9-b5e836a448f6" (UID: "d66c7a93-c12f-4185-8bf9-b5e836a448f6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:14.266992 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.266973    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-api-access-9qfzt" (OuterVolumeSpecName: "kube-api-access-9qfzt") pod "d66c7a93-c12f-4185-8bf9-b5e836a448f6" (UID: "d66c7a93-c12f-4185-8bf9-b5e836a448f6"). InnerVolumeSpecName "kube-api-access-9qfzt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:34:14.332265 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.332236    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"]
Jul 28 17:34:14.334331 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:34:14.334306    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f457ff_5ca7_41e9_b1a8_b85449e64d91.slice/crio-0b8d84a840ab00ccf1624f70481ff1789ebb451f7633565dc21bf6e1ae577dcc WatchSource:0}: Error finding container 0b8d84a840ab00ccf1624f70481ff1789ebb451f7633565dc21bf6e1ae577dcc: Status 404 returned error can't find the container with id 0b8d84a840ab00ccf1624f70481ff1789ebb451f7633565dc21bf6e1ae577dcc
Jul 28 17:34:14.365845 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.365824    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66c7a93-c12f-4185-8bf9-b5e836a448f6-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:34:14.365845 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.365845    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:34:14.365968 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.365857    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfzt\" (UniqueName: \"kubernetes.io/projected/d66c7a93-c12f-4185-8bf9-b5e836a448f6-kube-api-access-9qfzt\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:34:14.844264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.844224    2563 generic.go:356] "Generic (PLEG): container finished" podID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerID="fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7" exitCode=2
Jul 28 17:34:14.844264 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.844259    2563 generic.go:356] "Generic (PLEG): container finished" podID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerID="8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79" exitCode=2
Jul 28 17:34:14.844841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.844291    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"
Jul 28 17:34:14.844841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.844323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" event={"ID":"d66c7a93-c12f-4185-8bf9-b5e836a448f6","Type":"ContainerDied","Data":"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7"}
Jul 28 17:34:14.844841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.844407    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" event={"ID":"d66c7a93-c12f-4185-8bf9-b5e836a448f6","Type":"ContainerDied","Data":"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79"}
Jul 28 17:34:14.844841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.844441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8" event={"ID":"d66c7a93-c12f-4185-8bf9-b5e836a448f6","Type":"ContainerDied","Data":"74f67ee524630400cbe4cda8188b726777b417c727b322e97f5eec65e217a6bf"}
Jul 28 17:34:14.844841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.844483    2563 scope.go:117] "RemoveContainer" containerID="fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7"
Jul 28 17:34:14.847306 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.847282    2563 generic.go:356] "Generic (PLEG): container finished" podID="201ee33a-6197-4d86-90b2-0407348a27a6" containerID="b7a4a72188a8030538d8a340420550778b0c49937a7410375c0f25febbedb4cb" exitCode=2
Jul 28 17:34:14.847447 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.847355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerDied","Data":"b7a4a72188a8030538d8a340420550778b0c49937a7410375c0f25febbedb4cb"}
Jul 28 17:34:14.849060 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.849038    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" event={"ID":"95f457ff-5ca7-41e9-b1a8-b85449e64d91","Type":"ContainerStarted","Data":"07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61"}
Jul 28 17:34:14.849175 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.849100    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" event={"ID":"95f457ff-5ca7-41e9-b1a8-b85449e64d91","Type":"ContainerStarted","Data":"0b8d84a840ab00ccf1624f70481ff1789ebb451f7633565dc21bf6e1ae577dcc"}
Jul 28 17:34:14.863136 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.863116    2563 scope.go:117] "RemoveContainer" containerID="8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79"
Jul 28 17:34:14.871172 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.871155    2563 scope.go:117] "RemoveContainer" containerID="fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7"
Jul 28 17:34:14.871467 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:34:14.871447    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7\": container with ID starting with fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7 not found: ID does not exist" containerID="fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7"
Jul 28 17:34:14.871528 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.871479    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7"} err="failed to get container status \"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7\": rpc error: code = NotFound desc = could not find container \"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7\": container with ID starting with fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7 not found: ID does not exist"
Jul 28 17:34:14.871528 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.871496    2563 scope.go:117] "RemoveContainer" containerID="8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79"
Jul 28 17:34:14.871784 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:34:14.871766    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79\": container with ID starting with 8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79 not found: ID does not exist" containerID="8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79"
Jul 28 17:34:14.871836 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.871792    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79"} err="failed to get container status \"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79\": rpc error: code = NotFound desc = could not find container \"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79\": container with ID starting with 8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79 not found: ID does not exist"
Jul 28 17:34:14.871836 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.871811    2563 scope.go:117] "RemoveContainer" containerID="fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7"
Jul 28 17:34:14.872083 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.872066    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7"} err="failed to get container status \"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7\": rpc error: code = NotFound desc = could not find container \"fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7\": container with ID starting with fc0b184df9d4ea1a9f8fc3501a8566cb52b00eb589da0a3cb22a6644fec482b7 not found: ID does not exist"
Jul 28 17:34:14.872144 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.872085    2563 scope.go:117] "RemoveContainer" containerID="8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79"
Jul 28 17:34:14.872297 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.872282    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79"} err="failed to get container status \"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79\": rpc error: code = NotFound desc = could not find container \"8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79\": container with ID starting with 8596ad204adfafd35433ac04cbd683e23e62db32c8df05c95695c96f0f564c79 not found: ID does not exist"
Jul 28 17:34:14.882384 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.882364    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"]
Jul 28 17:34:14.886689 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:14.886653    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-raw-d84a6-predictor-6478888887-8rwd8"]
Jul 28 17:34:15.484091 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:15.484054    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:34:15.596185 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:15.596154    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" path="/var/lib/kubelet/pods/d66c7a93-c12f-4185-8bf9-b5e836a448f6/volumes"
Jul 28 17:34:18.868026 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:18.867997    2563 generic.go:356] "Generic (PLEG): container finished" podID="201ee33a-6197-4d86-90b2-0407348a27a6" containerID="cf7f96c52a843a7efbdc6f60ef6d253e60aefcdcd3f41a2b709b97be9596d916" exitCode=0
Jul 28 17:34:18.868381 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:18.868073    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerDied","Data":"cf7f96c52a843a7efbdc6f60ef6d253e60aefcdcd3f41a2b709b97be9596d916"}
Jul 28 17:34:18.869479 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:18.869457    2563 generic.go:356] "Generic (PLEG): container finished" podID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerID="07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61" exitCode=0
Jul 28 17:34:18.869593 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:18.869489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" event={"ID":"95f457ff-5ca7-41e9-b1a8-b85449e64d91","Type":"ContainerDied","Data":"07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61"}
Jul 28 17:34:19.876506 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:19.876470    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" event={"ID":"95f457ff-5ca7-41e9-b1a8-b85449e64d91","Type":"ContainerStarted","Data":"fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6"}
Jul 28 17:34:19.876915 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:19.876529    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" event={"ID":"95f457ff-5ca7-41e9-b1a8-b85449e64d91","Type":"ContainerStarted","Data":"c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6"}
Jul 28 17:34:19.876915 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:19.876781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:19.897436 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:19.897390    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podStartSLOduration=3.263053812 podStartE2EDuration="6.897375194s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.634321382s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:34:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:34:19.89574451 +0000 UTC m=+1036.829143928" watchObservedRunningTime="2026-07-28 17:34:19.897375194 +0000 UTC m=+1036.830774613"
Jul 28 17:34:20.484450 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:20.484409    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:34:20.487738 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:20.487704    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:34:20.488080 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:20.488053    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:20.881145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:20.881109    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:20.882131 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:20.882105    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:34:21.885073 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:21.885030    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:34:25.484112 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:25.484071    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:34:25.484473 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:25.484203    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:34:26.889177 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:26.889148    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:34:26.889733 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:26.889705    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:34:30.484144 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:30.484102    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:34:30.487455 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:30.487426    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:34:30.487801 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:30.487779    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:35.484070 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:35.484032    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:34:36.889609 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:36.889569    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:34:40.483649 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:40.483609    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 28 17:34:40.487937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:40.487906    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 28 17:34:40.488053 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:40.488039    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:34:40.488275 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:40.488238    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 28 17:34:40.488379 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:40.488369    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:34:43.979138 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:43.979109    2563 generic.go:356] "Generic (PLEG): container finished" podID="201ee33a-6197-4d86-90b2-0407348a27a6" containerID="63051469b8cb2ed0338862044e807f17cdde8e17b42a7ea348cd36ad45ee8b94" exitCode=0
Jul 28 17:34:43.979539 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:43.979158    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerDied","Data":"63051469b8cb2ed0338862044e807f17cdde8e17b42a7ea348cd36ad45ee8b94"}
Jul 28 17:34:43.979539 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:43.979210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v" event={"ID":"201ee33a-6197-4d86-90b2-0407348a27a6","Type":"ContainerDied","Data":"fc7b68c70cbe7a7e52920146b5265e6ca1a8820ae34b786fe546a7e7ac2e264d"}
Jul 28 17:34:43.979539 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:43.979234    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc7b68c70cbe7a7e52920146b5265e6ca1a8820ae34b786fe546a7e7ac2e264d"
Jul 28 17:34:43.985518 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:43.985498    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:34:44.083582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.083556    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm8v7\" (UniqueName: \"kubernetes.io/projected/201ee33a-6197-4d86-90b2-0407348a27a6-kube-api-access-qm8v7\") pod \"201ee33a-6197-4d86-90b2-0407348a27a6\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") "
Jul 28 17:34:44.083740 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.083594    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls\") pod \"201ee33a-6197-4d86-90b2-0407348a27a6\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") "
Jul 28 17:34:44.083740 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.083614    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-cabundle-cert\") pod \"201ee33a-6197-4d86-90b2-0407348a27a6\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") "
Jul 28 17:34:44.083740 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.083650    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201ee33a-6197-4d86-90b2-0407348a27a6-kserve-provision-location\") pod \"201ee33a-6197-4d86-90b2-0407348a27a6\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") "
Jul 28 17:34:44.083740 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.083690    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-kube-rbac-proxy-sar-config\") pod \"201ee33a-6197-4d86-90b2-0407348a27a6\" (UID: \"201ee33a-6197-4d86-90b2-0407348a27a6\") "
Jul 28 17:34:44.084025 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.084001    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/201ee33a-6197-4d86-90b2-0407348a27a6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "201ee33a-6197-4d86-90b2-0407348a27a6" (UID: "201ee33a-6197-4d86-90b2-0407348a27a6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:34:44.084077 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.084058    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "201ee33a-6197-4d86-90b2-0407348a27a6" (UID: "201ee33a-6197-4d86-90b2-0407348a27a6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:44.084126 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.084105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "201ee33a-6197-4d86-90b2-0407348a27a6" (UID: "201ee33a-6197-4d86-90b2-0407348a27a6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:34:44.085587 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.085567    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "201ee33a-6197-4d86-90b2-0407348a27a6" (UID: "201ee33a-6197-4d86-90b2-0407348a27a6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:34:44.085693 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.085613    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/201ee33a-6197-4d86-90b2-0407348a27a6-kube-api-access-qm8v7" (OuterVolumeSpecName: "kube-api-access-qm8v7") pod "201ee33a-6197-4d86-90b2-0407348a27a6" (UID: "201ee33a-6197-4d86-90b2-0407348a27a6"). InnerVolumeSpecName "kube-api-access-qm8v7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:34:44.185254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.185203    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm8v7\" (UniqueName: \"kubernetes.io/projected/201ee33a-6197-4d86-90b2-0407348a27a6-kube-api-access-qm8v7\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:34:44.185254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.185223    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/201ee33a-6197-4d86-90b2-0407348a27a6-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:34:44.185254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.185233    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:34:44.185254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.185241    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/201ee33a-6197-4d86-90b2-0407348a27a6-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:34:44.185254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.185250    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/201ee33a-6197-4d86-90b2-0407348a27a6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:34:44.983299 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:44.983270    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"
Jul 28 17:34:45.006345 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:45.006323    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"]
Jul 28 17:34:45.009733 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:45.009711    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-logger-raw-d84a6-predictor-5686cd4dcd-lg44v"]
Jul 28 17:34:45.596235 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:45.596202    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" path="/var/lib/kubelet/pods/201ee33a-6197-4d86-90b2-0407348a27a6/volumes"
Jul 28 17:34:46.889971 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:46.889933    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:34:56.890397 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:34:56.890309    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:35:06.889991 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:35:06.889955    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:35:16.889837 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:35:16.889799    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:35:26.889757 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:35:26.889717    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:35:36.889830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:35:36.889789    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:35:44.592218 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:35:44.592176    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:35:54.592985 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:35:54.592942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:36:04.593036 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:04.592996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:36:14.592879 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:14.592836    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:36:24.592338 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:24.592296    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:36:34.592927 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:34.592889    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:36:44.592856 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:44.592825    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:36:54.036415 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:54.036380    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"]
Jul 28 17:36:54.036905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:54.036707    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" containerID="cri-o://c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6" gracePeriod=30
Jul 28 17:36:54.036905 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:54.036734    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kube-rbac-proxy" containerID="cri-o://fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6" gracePeriod=30
Jul 28 17:36:54.505773 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:54.505745    2563 generic.go:356] "Generic (PLEG): container finished" podID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerID="fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6" exitCode=2
Jul 28 17:36:54.505930 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:54.505827    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" event={"ID":"95f457ff-5ca7-41e9-b1a8-b85449e64d91","Type":"ContainerDied","Data":"fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6"}
Jul 28 17:36:54.592140 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:54.592110    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 28 17:36:56.885747 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:56.885707    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:36:59.197422 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197387    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"]
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197703    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerName="kube-rbac-proxy"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197714    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerName="kube-rbac-proxy"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197727    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="storage-initializer"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197732    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="storage-initializer"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197747    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197757    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197762    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197770    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerName="kserve-container"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197775    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerName="kserve-container"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197783    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197788    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197834    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="agent"
Jul 28 17:36:59.197834 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197841    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kube-rbac-proxy"
Jul 28 17:36:59.198273 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197850    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerName="kserve-container"
Jul 28 17:36:59.198273 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197855    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66c7a93-c12f-4185-8bf9-b5e836a448f6" containerName="kube-rbac-proxy"
Jul 28 17:36:59.198273 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.197862    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="201ee33a-6197-4d86-90b2-0407348a27a6" containerName="kserve-container"
Jul 28 17:36:59.201024 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.201008    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.203619 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.203595    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-kube-rbac-proxy-sar-config\""
Jul 28 17:36:59.203762 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.203661    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-predictor-serving-cert\""
Jul 28 17:36:59.207651 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.207615    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"]
Jul 28 17:36:59.211499 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.211479    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.212825 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.212805    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"]
Jul 28 17:36:59.213945 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.213923    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-create-v2-predictor-serving-cert\""
Jul 28 17:36:59.225714 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.225694    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"]
Jul 28 17:36:59.312851 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.312825    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3efce03-ba23-4e2f-9483-fdb751356cf5-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.312980 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.312854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.312980 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.312889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.312980 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.312963    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5v69s\" (UniqueName: \"kubernetes.io/projected/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-api-access-5v69s\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.313130 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.313013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.313130 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.313046    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.313130 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.313073    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3efce03-ba23-4e2f-9483-fdb751356cf5-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.313130 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.313087    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shqpv\" (UniqueName: \"kubernetes.io/projected/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-api-access-shqpv\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.313130 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.313104    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.313309 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.313141    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.414189 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414158    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.414399 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5v69s\" (UniqueName: \"kubernetes.io/projected/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-api-access-5v69s\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.414399 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.414399 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.414399 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414282    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3efce03-ba23-4e2f-9483-fdb751356cf5-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.414399 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414331    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shqpv\" (UniqueName: \"kubernetes.io/projected/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-api-access-shqpv\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.414399 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414377    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.414767 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.414767 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414468    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3efce03-ba23-4e2f-9483-fdb751356cf5-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.414767 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.414767 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kserve-provision-location\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.414977 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414916    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3efce03-ba23-4e2f-9483-fdb751356cf5-kserve-provision-location\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.415032 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.414986    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.415168 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.415143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-cabundle-cert\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.415168 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.415162    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-rbac-proxy-sar-config\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.415279 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.415175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-cabundle-cert\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.416817 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.416793    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3efce03-ba23-4e2f-9483-fdb751356cf5-proxy-tls\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.417007 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.416990    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-proxy-tls\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.422052 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.422032    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-shqpv\" (UniqueName: \"kubernetes.io/projected/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-api-access-shqpv\") pod \"isvc-canary-create-predictor-cb85cf55c-vx4z8\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") " pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.422140 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.422129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5v69s\" (UniqueName: \"kubernetes.io/projected/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-api-access-5v69s\") pod \"isvc-canary-create-v2-predictor-6d8d4768d4-kx95s\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") " pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.512094 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.512037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:36:59.521318 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.521300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:36:59.648619 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.648596    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"]
Jul 28 17:36:59.650605 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:36:59.650576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3efce03_ba23_4e2f_9483_fdb751356cf5.slice/crio-7889501310f83173cbc0e99d0b8edd6a36c42b7d2bc2c45b9d210f353786f33f WatchSource:0}: Error finding container 7889501310f83173cbc0e99d0b8edd6a36c42b7d2bc2c45b9d210f353786f33f: Status 404 returned error can't find the container with id 7889501310f83173cbc0e99d0b8edd6a36c42b7d2bc2c45b9d210f353786f33f
Jul 28 17:36:59.664616 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:36:59.664594    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"]
Jul 28 17:36:59.666740 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:36:59.666720    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28b24a2a_60d2_4fdd_a41b_f34b933f12a2.slice/crio-519a2a615c6e14a63ca18ccf598730806cbb500c2890d046d4fe0ebfed2bcaa3 WatchSource:0}: Error finding container 519a2a615c6e14a63ca18ccf598730806cbb500c2890d046d4fe0ebfed2bcaa3: Status 404 returned error can't find the container with id 519a2a615c6e14a63ca18ccf598730806cbb500c2890d046d4fe0ebfed2bcaa3
Jul 28 17:37:00.530534 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:00.530489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" event={"ID":"a3efce03-ba23-4e2f-9483-fdb751356cf5","Type":"ContainerStarted","Data":"ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e"}
Jul 28 17:37:00.530955 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:00.530571    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" event={"ID":"a3efce03-ba23-4e2f-9483-fdb751356cf5","Type":"ContainerStarted","Data":"7889501310f83173cbc0e99d0b8edd6a36c42b7d2bc2c45b9d210f353786f33f"}
Jul 28 17:37:00.532021 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:00.531997    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" event={"ID":"28b24a2a-60d2-4fdd-a41b-f34b933f12a2","Type":"ContainerStarted","Data":"2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd"}
Jul 28 17:37:00.532143 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:00.532050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" event={"ID":"28b24a2a-60d2-4fdd-a41b-f34b933f12a2","Type":"ContainerStarted","Data":"519a2a615c6e14a63ca18ccf598730806cbb500c2890d046d4fe0ebfed2bcaa3"}
Jul 28 17:37:01.885250 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:01.885208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 28 17:37:03.084841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.084819    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:37:03.142832 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.142808    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kserve-provision-location\") pod \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") "
Jul 28 17:37:03.142952 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.142852    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-cabundle-cert\") pod \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") "
Jul 28 17:37:03.142952 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.142872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-rbac-proxy-sar-config\") pod \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") "
Jul 28 17:37:03.142952 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.142924    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f457ff-5ca7-41e9-b1a8-b85449e64d91-proxy-tls\") pod \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") "
Jul 28 17:37:03.143114 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.143081    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "95f457ff-5ca7-41e9-b1a8-b85449e64d91" (UID: "95f457ff-5ca7-41e9-b1a8-b85449e64d91"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:37:03.143172 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.143159    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m5grm\" (UniqueName: \"kubernetes.io/projected/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-api-access-m5grm\") pod \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\" (UID: \"95f457ff-5ca7-41e9-b1a8-b85449e64d91\") "
Jul 28 17:37:03.143254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.143231    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "95f457ff-5ca7-41e9-b1a8-b85449e64d91" (UID: "95f457ff-5ca7-41e9-b1a8-b85449e64d91"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:03.143316 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.143280    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "95f457ff-5ca7-41e9-b1a8-b85449e64d91" (UID: "95f457ff-5ca7-41e9-b1a8-b85449e64d91"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:37:03.143426 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.143409    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:37:03.143479 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.143433    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:37:03.143479 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.143448    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:37:03.144917 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.144896    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-api-access-m5grm" (OuterVolumeSpecName: "kube-api-access-m5grm") pod "95f457ff-5ca7-41e9-b1a8-b85449e64d91" (UID: "95f457ff-5ca7-41e9-b1a8-b85449e64d91"). InnerVolumeSpecName "kube-api-access-m5grm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:37:03.145128 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.145110    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/95f457ff-5ca7-41e9-b1a8-b85449e64d91-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "95f457ff-5ca7-41e9-b1a8-b85449e64d91" (UID: "95f457ff-5ca7-41e9-b1a8-b85449e64d91"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:37:03.243804 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.243771    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5grm\" (UniqueName: \"kubernetes.io/projected/95f457ff-5ca7-41e9-b1a8-b85449e64d91-kube-api-access-m5grm\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:37:03.243804 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.243804    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/95f457ff-5ca7-41e9-b1a8-b85449e64d91-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:37:03.545143 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.545111    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerID="ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e" exitCode=0
Jul 28 17:37:03.545266 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.545187    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" event={"ID":"a3efce03-ba23-4e2f-9483-fdb751356cf5","Type":"ContainerDied","Data":"ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e"}
Jul 28 17:37:03.546613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.546583    2563 generic.go:356] "Generic (PLEG): container finished" podID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerID="2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd" exitCode=0
Jul 28 17:37:03.546746 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.546648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" event={"ID":"28b24a2a-60d2-4fdd-a41b-f34b933f12a2","Type":"ContainerDied","Data":"2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd"}
Jul 28 17:37:03.548553 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.548532    2563 generic.go:356] "Generic (PLEG): container finished" podID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerID="c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6" exitCode=0
Jul 28 17:37:03.548689 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.548572    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" event={"ID":"95f457ff-5ca7-41e9-b1a8-b85449e64d91","Type":"ContainerDied","Data":"c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6"}
Jul 28 17:37:03.548689 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.548594    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"
Jul 28 17:37:03.548689 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.548617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh" event={"ID":"95f457ff-5ca7-41e9-b1a8-b85449e64d91","Type":"ContainerDied","Data":"0b8d84a840ab00ccf1624f70481ff1789ebb451f7633565dc21bf6e1ae577dcc"}
Jul 28 17:37:03.548689 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.548643    2563 scope.go:117] "RemoveContainer" containerID="fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6"
Jul 28 17:37:03.561293 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.561275    2563 scope.go:117] "RemoveContainer" containerID="c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6"
Jul 28 17:37:03.571485 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.571458    2563 scope.go:117] "RemoveContainer" containerID="07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61"
Jul 28 17:37:03.581191 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.581172    2563 scope.go:117] "RemoveContainer" containerID="fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6"
Jul 28 17:37:03.581522 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:37:03.581496    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6\": container with ID starting with fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6 not found: ID does not exist" containerID="fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6"
Jul 28 17:37:03.581587 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.581535    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6"} err="failed to get container status \"fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6\": rpc error: code = NotFound desc = could not find container \"fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6\": container with ID starting with fa3b1129c62aae3f919ceaa286ec02d2d3e1eb7ca58c03a894ed0cbdf85601a6 not found: ID does not exist"
Jul 28 17:37:03.581587 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.581563    2563 scope.go:117] "RemoveContainer" containerID="c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6"
Jul 28 17:37:03.581874 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:37:03.581849    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6\": container with ID starting with c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6 not found: ID does not exist" containerID="c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6"
Jul 28 17:37:03.581984 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.581878    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6"} err="failed to get container status \"c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6\": rpc error: code = NotFound desc = could not find container \"c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6\": container with ID starting with c07852cd0016733599b74fccd90a71914575244aed7e212e87468e5786f361b6 not found: ID does not exist"
Jul 28 17:37:03.581984 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.581893    2563 scope.go:117] "RemoveContainer" containerID="07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61"
Jul 28 17:37:03.582133 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:37:03.582115    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61\": container with ID starting with 07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61 not found: ID does not exist" containerID="07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61"
Jul 28 17:37:03.582195 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.582137    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61"} err="failed to get container status \"07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61\": rpc error: code = NotFound desc = could not find container \"07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61\": container with ID starting with 07c2ab243f2f8c6f7fdca043429773a43a77ffeaf2cc7c185ab5714d2c6fbb61 not found: ID does not exist"
Jul 28 17:37:03.600613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.600593    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"]
Jul 28 17:37:03.600794 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:03.600773    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-scale-raw-dca0e-predictor-6d97785746-tzlgh"]
Jul 28 17:37:04.557660 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.557625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" event={"ID":"a3efce03-ba23-4e2f-9483-fdb751356cf5","Type":"ContainerStarted","Data":"300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb"}
Jul 28 17:37:04.558123 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.557716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" event={"ID":"a3efce03-ba23-4e2f-9483-fdb751356cf5","Type":"ContainerStarted","Data":"bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec"}
Jul 28 17:37:04.558123 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.558034    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:37:04.558123 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.558108    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:37:04.559215 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.559189    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:04.559788 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.559767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" event={"ID":"28b24a2a-60d2-4fdd-a41b-f34b933f12a2","Type":"ContainerStarted","Data":"04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e"}
Jul 28 17:37:04.559875 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.559826    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" event={"ID":"28b24a2a-60d2-4fdd-a41b-f34b933f12a2","Type":"ContainerStarted","Data":"c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af"}
Jul 28 17:37:04.560056 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.560040    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:37:04.578537 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.578499    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podStartSLOduration=1.789745948 podStartE2EDuration="5.57848523s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.788739282s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:04.577471661 +0000 UTC m=+1201.510871080" watchObservedRunningTime="2026-07-28 17:37:04.57848523 +0000 UTC m=+1201.511884711"
Jul 28 17:37:04.597227 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:04.597180    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podStartSLOduration=1.857333778 podStartE2EDuration="5.597168341s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.739834563s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:36:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:37:04.59617633 +0000 UTC m=+1201.529575753" watchObservedRunningTime="2026-07-28 17:37:04.597168341 +0000 UTC m=+1201.530567759"
Jul 28 17:37:05.565235 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:05.565200    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:05.565609 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:05.565275    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:37:05.566245 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:05.566220    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:37:05.597012 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:05.596980    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" path="/var/lib/kubelet/pods/95f457ff-5ca7-41e9-b1a8-b85449e64d91/volumes"
Jul 28 17:37:06.569751 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:06.569709    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:37:10.569227 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:10.569201    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:37:10.569729 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:10.569701    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:11.574292 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:11.574264    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:37:11.574787 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:11.574762    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:37:20.570577 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:20.570539    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:21.575601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:21.575565    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:37:30.569972 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:30.569932    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:31.574995 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:31.574956    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:37:40.569771 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:40.569728    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:41.575249 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:41.575213    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:37:50.569776 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:50.569642    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:37:51.575141 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:37:51.575098    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:38:00.570035 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:00.569945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:38:01.574746 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:01.574709    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:38:10.569622 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:10.569579    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:38:11.575488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:11.575446    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:38:19.596535 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:19.596507    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:38:20.570379 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:20.570352    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:38:29.305307 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.305276    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"]
Jul 28 17:38:29.305765 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.305509    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" containerID="cri-o://c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af" gracePeriod=30
Jul 28 17:38:29.305765 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.305543    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kube-rbac-proxy" containerID="cri-o://04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e" gracePeriod=30
Jul 28 17:38:29.309152 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.309127    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"]
Jul 28 17:38:29.309430 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.309407    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" containerID="cri-o://bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec" gracePeriod=30
Jul 28 17:38:29.309488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.309426    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kube-rbac-proxy" containerID="cri-o://300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb" gracePeriod=30
Jul 28 17:38:29.593169 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.593128    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 28 17:38:29.899464 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.899372    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerID="300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb" exitCode=2
Jul 28 17:38:29.899464 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.899443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" event={"ID":"a3efce03-ba23-4e2f-9483-fdb751356cf5","Type":"ContainerDied","Data":"300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb"}
Jul 28 17:38:29.901708 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.901686    2563 generic.go:356] "Generic (PLEG): container finished" podID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerID="04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e" exitCode=2
Jul 28 17:38:29.901840 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:29.901769    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" event={"ID":"28b24a2a-60d2-4fdd-a41b-f34b933f12a2","Type":"ContainerDied","Data":"04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e"}
Jul 28 17:38:30.565749 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:30.565699    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 28 17:38:30.569995 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:30.569972    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 28 17:38:31.569875 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:31.569836    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 28 17:38:33.563979 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.563951    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:38:33.566526 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.566510    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:38:33.649455 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649424    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kserve-provision-location\") pod \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") "
Jul 28 17:38:33.649613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649478    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-cabundle-cert\") pod \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") "
Jul 28 17:38:33.649613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649495    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-rbac-proxy-sar-config\") pod \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") "
Jul 28 17:38:33.649613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649514    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3efce03-ba23-4e2f-9483-fdb751356cf5-proxy-tls\") pod \"a3efce03-ba23-4e2f-9483-fdb751356cf5\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") "
Jul 28 17:38:33.649613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649531    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-shqpv\" (UniqueName: \"kubernetes.io/projected/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-api-access-shqpv\") pod \"a3efce03-ba23-4e2f-9483-fdb751356cf5\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") "
Jul 28 17:38:33.649613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649565    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5v69s\" (UniqueName: \"kubernetes.io/projected/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-api-access-5v69s\") pod \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") "
Jul 28 17:38:33.649909 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-cabundle-cert\") pod \"a3efce03-ba23-4e2f-9483-fdb751356cf5\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") "
Jul 28 17:38:33.649909 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649803    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3efce03-ba23-4e2f-9483-fdb751356cf5-kserve-provision-location\") pod \"a3efce03-ba23-4e2f-9483-fdb751356cf5\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") "
Jul 28 17:38:33.649909 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649837    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-proxy-tls\") pod \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\" (UID: \"28b24a2a-60d2-4fdd-a41b-f34b933f12a2\") "
Jul 28 17:38:33.649909 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649777    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28b24a2a-60d2-4fdd-a41b-f34b933f12a2" (UID: "28b24a2a-60d2-4fdd-a41b-f34b933f12a2"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:33.649909 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649864    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28b24a2a-60d2-4fdd-a41b-f34b933f12a2" (UID: "28b24a2a-60d2-4fdd-a41b-f34b933f12a2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:33.649909 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-rbac-proxy-sar-config\") pod \"a3efce03-ba23-4e2f-9483-fdb751356cf5\" (UID: \"a3efce03-ba23-4e2f-9483-fdb751356cf5\") "
Jul 28 17:38:33.650228 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.649918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28b24a2a-60d2-4fdd-a41b-f34b933f12a2" (UID: "28b24a2a-60d2-4fdd-a41b-f34b933f12a2"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:33.650228 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.650149    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.650228 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.650170    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.650228 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.650186    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.650228 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.650097    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3efce03-ba23-4e2f-9483-fdb751356cf5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "a3efce03-ba23-4e2f-9483-fdb751356cf5" (UID: "a3efce03-ba23-4e2f-9483-fdb751356cf5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:38:33.650492 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.650261    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "a3efce03-ba23-4e2f-9483-fdb751356cf5" (UID: "a3efce03-ba23-4e2f-9483-fdb751356cf5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:33.650492 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.650345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a3efce03-ba23-4e2f-9483-fdb751356cf5" (UID: "a3efce03-ba23-4e2f-9483-fdb751356cf5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:38:33.651695 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.651628    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-api-access-shqpv" (OuterVolumeSpecName: "kube-api-access-shqpv") pod "a3efce03-ba23-4e2f-9483-fdb751356cf5" (UID: "a3efce03-ba23-4e2f-9483-fdb751356cf5"). InnerVolumeSpecName "kube-api-access-shqpv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:33.651695 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.651638    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3efce03-ba23-4e2f-9483-fdb751356cf5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a3efce03-ba23-4e2f-9483-fdb751356cf5" (UID: "a3efce03-ba23-4e2f-9483-fdb751356cf5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:33.651852 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.651834    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28b24a2a-60d2-4fdd-a41b-f34b933f12a2" (UID: "28b24a2a-60d2-4fdd-a41b-f34b933f12a2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:38:33.651962 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.651943    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-api-access-5v69s" (OuterVolumeSpecName: "kube-api-access-5v69s") pod "28b24a2a-60d2-4fdd-a41b-f34b933f12a2" (UID: "28b24a2a-60d2-4fdd-a41b-f34b933f12a2"). InnerVolumeSpecName "kube-api-access-5v69s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:38:33.750880 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.750860    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5v69s\" (UniqueName: \"kubernetes.io/projected/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-kube-api-access-5v69s\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.750961 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.750901    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.750961 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.750949    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/a3efce03-ba23-4e2f-9483-fdb751356cf5-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.751032 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.750968    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28b24a2a-60d2-4fdd-a41b-f34b933f12a2-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.751032 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.750980    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.751032 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.750989    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a3efce03-ba23-4e2f-9483-fdb751356cf5-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.751032 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.751000    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shqpv\" (UniqueName: \"kubernetes.io/projected/a3efce03-ba23-4e2f-9483-fdb751356cf5-kube-api-access-shqpv\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:38:33.919915 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.919851    2563 generic.go:356] "Generic (PLEG): container finished" podID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerID="bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec" exitCode=0
Jul 28 17:38:33.919915 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.919907    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"
Jul 28 17:38:33.920068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.919936    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" event={"ID":"a3efce03-ba23-4e2f-9483-fdb751356cf5","Type":"ContainerDied","Data":"bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec"}
Jul 28 17:38:33.920068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.919999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8" event={"ID":"a3efce03-ba23-4e2f-9483-fdb751356cf5","Type":"ContainerDied","Data":"7889501310f83173cbc0e99d0b8edd6a36c42b7d2bc2c45b9d210f353786f33f"}
Jul 28 17:38:33.920068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.920036    2563 scope.go:117] "RemoveContainer" containerID="300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb"
Jul 28 17:38:33.921863 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.921840    2563 generic.go:356] "Generic (PLEG): container finished" podID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerID="c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af" exitCode=0
Jul 28 17:38:33.921966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.921900    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"
Jul 28 17:38:33.921966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.921908    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" event={"ID":"28b24a2a-60d2-4fdd-a41b-f34b933f12a2","Type":"ContainerDied","Data":"c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af"}
Jul 28 17:38:33.921966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.921948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s" event={"ID":"28b24a2a-60d2-4fdd-a41b-f34b933f12a2","Type":"ContainerDied","Data":"519a2a615c6e14a63ca18ccf598730806cbb500c2890d046d4fe0ebfed2bcaa3"}
Jul 28 17:38:33.927871 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.927855    2563 scope.go:117] "RemoveContainer" containerID="bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec"
Jul 28 17:38:33.935582 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.935565    2563 scope.go:117] "RemoveContainer" containerID="ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e"
Jul 28 17:38:33.942504 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.942488    2563 scope.go:117] "RemoveContainer" containerID="300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb"
Jul 28 17:38:33.942744 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:38:33.942727    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb\": container with ID starting with 300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb not found: ID does not exist" containerID="300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb"
Jul 28 17:38:33.942786 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.942752    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb"} err="failed to get container status \"300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb\": rpc error: code = NotFound desc = could not find container \"300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb\": container with ID starting with 300d9ae88ff97965eb15cc53aae3830fc8d0fa102eacabd55741548065a5d8fb not found: ID does not exist"
Jul 28 17:38:33.942786 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.942770    2563 scope.go:117] "RemoveContainer" containerID="bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec"
Jul 28 17:38:33.943008 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:38:33.942991    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec\": container with ID starting with bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec not found: ID does not exist" containerID="bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec"
Jul 28 17:38:33.943056 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.943014    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec"} err="failed to get container status \"bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec\": rpc error: code = NotFound desc = could not find container \"bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec\": container with ID starting with bbe3ae2d2b60235e4edb5691f9c260dd2dde244a37ba57213a6f4b4eac5428ec not found: ID does not exist"
Jul 28 17:38:33.943056 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.943029    2563 scope.go:117] "RemoveContainer" containerID="ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e"
Jul 28 17:38:33.943255 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:38:33.943238    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e\": container with ID starting with ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e not found: ID does not exist" containerID="ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e"
Jul 28 17:38:33.943294 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.943259    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e"} err="failed to get container status \"ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e\": rpc error: code = NotFound desc = could not find container \"ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e\": container with ID starting with ffe8eecee2b86257fd37164b370890635dc771fd34c78ff5609e5ac0eb93525e not found: ID does not exist"
Jul 28 17:38:33.943294 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.943273    2563 scope.go:117] "RemoveContainer" containerID="04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e"
Jul 28 17:38:33.946132 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.946110    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"]
Jul 28 17:38:33.949685 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.949656    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-predictor-cb85cf55c-vx4z8"]
Jul 28 17:38:33.950574 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.950561    2563 scope.go:117] "RemoveContainer" containerID="c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af"
Jul 28 17:38:33.957252 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.957238    2563 scope.go:117] "RemoveContainer" containerID="2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd"
Jul 28 17:38:33.961138 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.961118    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"]
Jul 28 17:38:33.964504 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.964489    2563 scope.go:117] "RemoveContainer" containerID="04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e"
Jul 28 17:38:33.964777 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:38:33.964751    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e\": container with ID starting with 04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e not found: ID does not exist" containerID="04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e"
Jul 28 17:38:33.964877 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.964784    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e"} err="failed to get container status \"04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e\": rpc error: code = NotFound desc = could not find container \"04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e\": container with ID starting with 04bc9a98bb02fb2168a6b695ac35dc61a0ec2f14a7e2b35957177039d2fa114e not found: ID does not exist"
Jul 28 17:38:33.964877 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.964804    2563 scope.go:117] "RemoveContainer" containerID="c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af"
Jul 28 17:38:33.964993 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.964947    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-create-v2-predictor-6d8d4768d4-kx95s"]
Jul 28 17:38:33.965051 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:38:33.965032    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af\": container with ID starting with c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af not found: ID does not exist" containerID="c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af"
Jul 28 17:38:33.965086 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.965056    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af"} err="failed to get container status \"c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af\": rpc error: code = NotFound desc = could not find container \"c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af\": container with ID starting with c0ed0dab7fb080020aaeaf5910ba20523692bcc342ec43810e6723f25cbc04af not found: ID does not exist"
Jul 28 17:38:33.965086 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.965070    2563 scope.go:117] "RemoveContainer" containerID="2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd"
Jul 28 17:38:33.965301 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:38:33.965285    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd\": container with ID starting with 2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd not found: ID does not exist" containerID="2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd"
Jul 28 17:38:33.965350 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:33.965305    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd"} err="failed to get container status \"2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd\": rpc error: code = NotFound desc = could not find container \"2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd\": container with ID starting with 2f8a962615cc0c7f7668ae6149c42eae077fcd15405d9a57bc61527c25e117fd not found: ID does not exist"
Jul 28 17:38:34.488710 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.488666    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"]
Jul 28 17:38:34.488992 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.488980    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.488993    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489003    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container"
Jul 28 17:38:34.489041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489009    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container"
Jul 28 17:38:34.489041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489017    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489023    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489032    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="storage-initializer"
Jul 28 17:38:34.489041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489037    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="storage-initializer"
Jul 28 17:38:34.489041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489043    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489049    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489056    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489062    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489070    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="storage-initializer"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489075    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="storage-initializer"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489081    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489086    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489092    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="storage-initializer"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489097    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="storage-initializer"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489145    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kserve-container"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489153    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489159    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489165    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" containerName="kube-rbac-proxy"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489170    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="95f457ff-5ca7-41e9-b1a8-b85449e64d91" containerName="kserve-container"
Jul 28 17:38:34.489315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.489175    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" containerName="kserve-container"
Jul 28 17:38:34.498436 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.498404    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"]
Jul 28 17:38:34.498539 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.498521    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.501587 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.501561    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:38:34.501721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.501587    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 28 17:38:34.501721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.501611    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:38:34.501721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.501688    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:38:34.501901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.501767    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.501901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.501821    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-predictor-serving-cert\""
Jul 28 17:38:34.502005 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.501930    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jwrfk\""
Jul 28 17:38:34.502067 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.502002    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:38:34.503644 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.503627    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"]
Jul 28 17:38:34.504698 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.504663    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 28 17:38:34.507310 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.507272    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"]
Jul 28 17:38:34.659705 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.659705 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659651    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4m8m8\" (UniqueName: \"kubernetes.io/projected/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-api-access-4m8m8\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.659705 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659693    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23616cc5-959c-48e4-8651-b8fa86fecc0c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.660097 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.660097 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.660097 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.660097 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23616cc5-959c-48e4-8651-b8fa86fecc0c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.660097 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659882    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnhl4\" (UniqueName: \"kubernetes.io/projected/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-api-access-tnhl4\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.660097 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.660097 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.659952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9baed3-9ba9-4189-ad81-18e61327ab86-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.760430 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760403    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9baed3-9ba9-4189-ad81-18e61327ab86-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.760546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.760546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4m8m8\" (UniqueName: \"kubernetes.io/projected/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-api-access-4m8m8\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.760546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760479    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23616cc5-959c-48e4-8651-b8fa86fecc0c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.760708 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.760760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760740    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.760799 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.760863 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23616cc5-959c-48e4-8651-b8fa86fecc0c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.760932 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:38:34.760862    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-promote-predictor-serving-cert: secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 17:38:34.760932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnhl4\" (UniqueName: \"kubernetes.io/projected/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-api-access-tnhl4\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.760932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760913    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.760932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.760923    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9baed3-9ba9-4189-ad81-18e61327ab86-kserve-provision-location\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.761135 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:38:34.760931    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls podName:fa9baed3-9ba9-4189-ad81-18e61327ab86 nodeName:}" failed. No retries permitted until 2026-07-28 17:38:35.260910527 +0000 UTC m=+1292.194309926 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls") pod "isvc-canary-promote-predictor-d9759ffd5-g8nsd" (UID: "fa9baed3-9ba9-4189-ad81-18e61327ab86") : secret "isvc-canary-promote-predictor-serving-cert" not found
Jul 28 17:38:34.761290 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.761266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.761379 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.761266    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23616cc5-959c-48e4-8651-b8fa86fecc0c-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.761379 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.761356    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-cabundle-cert\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.761500 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.761469    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.761500 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.761493    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.763307 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.763286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23616cc5-959c-48e4-8651-b8fa86fecc0c-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.771068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.771048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4m8m8\" (UniqueName: \"kubernetes.io/projected/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-api-access-4m8m8\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:34.771151 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.771052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnhl4\" (UniqueName: \"kubernetes.io/projected/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-api-access-tnhl4\") pod \"isvc-canary-promote-v2-predictor-86766c4958-djmzl\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.817115 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.817093    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:34.943534 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:34.943511    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"]
Jul 28 17:38:34.945474 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:38:34.945449    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23616cc5_959c_48e4_8651_b8fa86fecc0c.slice/crio-9ca5bac9072a798f7f881680848ad8630321f3649e6ab189fbeba0367daaccff WatchSource:0}: Error finding container 9ca5bac9072a798f7f881680848ad8630321f3649e6ab189fbeba0367daaccff: Status 404 returned error can't find the container with id 9ca5bac9072a798f7f881680848ad8630321f3649e6ab189fbeba0367daaccff
Jul 28 17:38:35.265364 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.265276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:35.267540 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.267519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls\") pod \"isvc-canary-promote-predictor-d9759ffd5-g8nsd\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") " pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:35.411210 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.411187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:35.540256 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.540228    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"]
Jul 28 17:38:35.542602 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:38:35.542576    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa9baed3_9ba9_4189_ad81_18e61327ab86.slice/crio-ea471d1d80de62e03e2cf4d79bc2ff1f9d65667376b6da2f7c1ec46e1edc7a45 WatchSource:0}: Error finding container ea471d1d80de62e03e2cf4d79bc2ff1f9d65667376b6da2f7c1ec46e1edc7a45: Status 404 returned error can't find the container with id ea471d1d80de62e03e2cf4d79bc2ff1f9d65667376b6da2f7c1ec46e1edc7a45
Jul 28 17:38:35.595912 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.595889    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28b24a2a-60d2-4fdd-a41b-f34b933f12a2" path="/var/lib/kubelet/pods/28b24a2a-60d2-4fdd-a41b-f34b933f12a2/volumes"
Jul 28 17:38:35.596360 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.596346    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3efce03-ba23-4e2f-9483-fdb751356cf5" path="/var/lib/kubelet/pods/a3efce03-ba23-4e2f-9483-fdb751356cf5/volumes"
Jul 28 17:38:35.934452 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.934411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" event={"ID":"fa9baed3-9ba9-4189-ad81-18e61327ab86","Type":"ContainerStarted","Data":"f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323"}
Jul 28 17:38:35.934878 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.934482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" event={"ID":"fa9baed3-9ba9-4189-ad81-18e61327ab86","Type":"ContainerStarted","Data":"ea471d1d80de62e03e2cf4d79bc2ff1f9d65667376b6da2f7c1ec46e1edc7a45"}
Jul 28 17:38:35.935996 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.935969    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" event={"ID":"23616cc5-959c-48e4-8651-b8fa86fecc0c","Type":"ContainerStarted","Data":"ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825"}
Jul 28 17:38:35.936093 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:35.936029    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" event={"ID":"23616cc5-959c-48e4-8651-b8fa86fecc0c","Type":"ContainerStarted","Data":"9ca5bac9072a798f7f881680848ad8630321f3649e6ab189fbeba0367daaccff"}
Jul 28 17:38:38.955221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:38.955135    2563 generic.go:356] "Generic (PLEG): container finished" podID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerID="ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825" exitCode=0
Jul 28 17:38:38.955221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:38.955206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" event={"ID":"23616cc5-959c-48e4-8651-b8fa86fecc0c","Type":"ContainerDied","Data":"ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825"}
Jul 28 17:38:39.961323 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:39.961285    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" event={"ID":"23616cc5-959c-48e4-8651-b8fa86fecc0c","Type":"ContainerStarted","Data":"fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3"}
Jul 28 17:38:39.961760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:39.961355    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" event={"ID":"23616cc5-959c-48e4-8651-b8fa86fecc0c","Type":"ContainerStarted","Data":"100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168"}
Jul 28 17:38:39.961760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:39.961560    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:39.962847 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:39.962822    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerID="f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323" exitCode=0
Jul 28 17:38:39.962946 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:39.962857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" event={"ID":"fa9baed3-9ba9-4189-ad81-18e61327ab86","Type":"ContainerDied","Data":"f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323"}
Jul 28 17:38:39.983999 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:39.983960    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podStartSLOduration=2.283420297 podStartE2EDuration="5.983946587s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.70052629s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:39.981596028 +0000 UTC m=+1296.914995447" watchObservedRunningTime="2026-07-28 17:38:39.983946587 +0000 UTC m=+1296.917346004"
Jul 28 17:38:40.969167 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:40.969127    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" event={"ID":"fa9baed3-9ba9-4189-ad81-18e61327ab86","Type":"ContainerStarted","Data":"edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d"}
Jul 28 17:38:40.969593 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:40.969198    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" event={"ID":"fa9baed3-9ba9-4189-ad81-18e61327ab86","Type":"ContainerStarted","Data":"c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12"}
Jul 28 17:38:40.969593 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:40.969426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:40.969733 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:40.969699    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:40.969861 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:40.969837    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:40.970640 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:40.970620    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:38:40.970721 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:40.970703    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:38:40.993069 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:40.993033    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podStartSLOduration=3.212537449 podStartE2EDuration="6.993020367s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.780482918s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:38:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:38:40.992474113 +0000 UTC m=+1297.925873536" watchObservedRunningTime="2026-07-28 17:38:40.993020367 +0000 UTC m=+1297.926419787"
Jul 28 17:38:41.973953 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:41.973908    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:38:41.974302 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:41.974002    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:38:46.977970 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:46.977936    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:38:46.978388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:46.978293    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:38:46.978430 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:46.978386    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:38:46.978761 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:46.978732    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:38:56.978747 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:56.978710    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:38:56.978747 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:38:56.978727    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:39:03.611032 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:03.611002    2563 scope.go:117] "RemoveContainer" containerID="63051469b8cb2ed0338862044e807f17cdde8e17b42a7ea348cd36ad45ee8b94"
Jul 28 17:39:03.618823 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:03.618799    2563 scope.go:117] "RemoveContainer" containerID="cf7f96c52a843a7efbdc6f60ef6d253e60aefcdcd3f41a2b709b97be9596d916"
Jul 28 17:39:03.628907 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:03.628883    2563 scope.go:117] "RemoveContainer" containerID="bce05794d261383601a47675aa9ec3429af566eba71d3b6f84d0749abd6d4af0"
Jul 28 17:39:03.636331 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:03.636314    2563 scope.go:117] "RemoveContainer" containerID="b7a4a72188a8030538d8a340420550778b0c49937a7410375c0f25febbedb4cb"
Jul 28 17:39:06.979359 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:06.979324    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:39:06.979753 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:06.979329    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:39:16.978953 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:16.978912    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:39:16.979321 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:16.978913    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:39:26.979127 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:26.979042    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:39:26.979127 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:26.979042    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:39:36.978417 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:36.978363    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:39:36.978916 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:36.978644    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:39:46.978822 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:46.978668    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 28 17:39:46.979345 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:46.978787    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:39:53.596097 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:53.596067    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:39:56.979833 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:56.979800    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:39:59.481941 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:59.481908    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"]
Jul 28 17:39:59.482316 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:59.482150    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" containerID="cri-o://c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12" gracePeriod=30
Jul 28 17:39:59.482316 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:39:59.482173    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kube-rbac-proxy" containerID="cri-o://edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d" gracePeriod=30
Jul 28 17:40:00.282226 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:00.282194    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerID="edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d" exitCode=2
Jul 28 17:40:00.282389 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:00.282246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" event={"ID":"fa9baed3-9ba9-4189-ad81-18e61327ab86","Type":"ContainerDied","Data":"edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d"}
Jul 28 17:40:01.974635 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:01.974591    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 28 17:40:03.593934 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.593870    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 28 17:40:03.722838 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.722816    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:40:03.752407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9baed3-9ba9-4189-ad81-18e61327ab86-kserve-provision-location\") pod \"fa9baed3-9ba9-4189-ad81-18e61327ab86\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") "
Jul 28 17:40:03.752407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752388    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-rbac-proxy-sar-config\") pod \"fa9baed3-9ba9-4189-ad81-18e61327ab86\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") "
Jul 28 17:40:03.752558 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752423    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-cabundle-cert\") pod \"fa9baed3-9ba9-4189-ad81-18e61327ab86\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") "
Jul 28 17:40:03.752558 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752503    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls\") pod \"fa9baed3-9ba9-4189-ad81-18e61327ab86\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") "
Jul 28 17:40:03.752558 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752537    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4m8m8\" (UniqueName: \"kubernetes.io/projected/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-api-access-4m8m8\") pod \"fa9baed3-9ba9-4189-ad81-18e61327ab86\" (UID: \"fa9baed3-9ba9-4189-ad81-18e61327ab86\") "
Jul 28 17:40:03.752754 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752692    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa9baed3-9ba9-4189-ad81-18e61327ab86-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "fa9baed3-9ba9-4189-ad81-18e61327ab86" (UID: "fa9baed3-9ba9-4189-ad81-18e61327ab86"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:40:03.752817 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752793    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "fa9baed3-9ba9-4189-ad81-18e61327ab86" (UID: "fa9baed3-9ba9-4189-ad81-18e61327ab86"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:03.752875 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752867    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/fa9baed3-9ba9-4189-ad81-18e61327ab86-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:40:03.752935 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.752880    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:40:03.753132 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.753108    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "fa9baed3-9ba9-4189-ad81-18e61327ab86" (UID: "fa9baed3-9ba9-4189-ad81-18e61327ab86"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:40:03.754585 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.754559    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-api-access-4m8m8" (OuterVolumeSpecName: "kube-api-access-4m8m8") pod "fa9baed3-9ba9-4189-ad81-18e61327ab86" (UID: "fa9baed3-9ba9-4189-ad81-18e61327ab86"). InnerVolumeSpecName "kube-api-access-4m8m8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:40:03.754666 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.754586    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "fa9baed3-9ba9-4189-ad81-18e61327ab86" (UID: "fa9baed3-9ba9-4189-ad81-18e61327ab86"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:40:03.853206 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.853181    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/fa9baed3-9ba9-4189-ad81-18e61327ab86-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:40:03.853307 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.853208    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4m8m8\" (UniqueName: \"kubernetes.io/projected/fa9baed3-9ba9-4189-ad81-18e61327ab86-kube-api-access-4m8m8\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:40:03.853307 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:03.853223    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/fa9baed3-9ba9-4189-ad81-18e61327ab86-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:40:04.299294 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.299262    2563 generic.go:356] "Generic (PLEG): container finished" podID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerID="c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12" exitCode=0
Jul 28 17:40:04.299475 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.299328    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"
Jul 28 17:40:04.299475 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.299350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" event={"ID":"fa9baed3-9ba9-4189-ad81-18e61327ab86","Type":"ContainerDied","Data":"c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12"}
Jul 28 17:40:04.299475 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.299424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd" event={"ID":"fa9baed3-9ba9-4189-ad81-18e61327ab86","Type":"ContainerDied","Data":"ea471d1d80de62e03e2cf4d79bc2ff1f9d65667376b6da2f7c1ec46e1edc7a45"}
Jul 28 17:40:04.299475 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.299462    2563 scope.go:117] "RemoveContainer" containerID="edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d"
Jul 28 17:40:04.308301 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.308281    2563 scope.go:117] "RemoveContainer" containerID="c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12"
Jul 28 17:40:04.315621 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.315606    2563 scope.go:117] "RemoveContainer" containerID="f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323"
Jul 28 17:40:04.322788 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.322764    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"]
Jul 28 17:40:04.323219 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.323194    2563 scope.go:117] "RemoveContainer" containerID="edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d"
Jul 28 17:40:04.323464 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:40:04.323446    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d\": container with ID starting with edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d not found: ID does not exist" containerID="edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d"
Jul 28 17:40:04.323538 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.323476    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d"} err="failed to get container status \"edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d\": rpc error: code = NotFound desc = could not find container \"edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d\": container with ID starting with edef045dd08d06dd94df6f9625b3e57a0e9e016dde8a3c8f5c6a5984c14f572d not found: ID does not exist"
Jul 28 17:40:04.323538 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.323500    2563 scope.go:117] "RemoveContainer" containerID="c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12"
Jul 28 17:40:04.323779 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:40:04.323757    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12\": container with ID starting with c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12 not found: ID does not exist" containerID="c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12"
Jul 28 17:40:04.323828 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.323788    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12"} err="failed to get container status \"c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12\": rpc error: code = NotFound desc = could not find container \"c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12\": container with ID starting with c683fb0bb1ff6f7ce4b1c1a0d6b0da2f8e3d3e46f473c29f593fdb7dca8d6d12 not found: ID does not exist"
Jul 28 17:40:04.323828 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.323811    2563 scope.go:117] "RemoveContainer" containerID="f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323"
Jul 28 17:40:04.324040 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:40:04.324021    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323\": container with ID starting with f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323 not found: ID does not exist" containerID="f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323"
Jul 28 17:40:04.324094 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.324047    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323"} err="failed to get container status \"f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323\": rpc error: code = NotFound desc = could not find container \"f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323\": container with ID starting with f8ae5367125549da63aa8e86e56f16b04dcd3e2375f9b384f58aa4b0c0cbf323 not found: ID does not exist"
Jul 28 17:40:04.324524 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:04.324507    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-predictor-d9759ffd5-g8nsd"]
Jul 28 17:40:05.595733 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:40:05.595699    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" path="/var/lib/kubelet/pods/fa9baed3-9ba9-4189-ad81-18e61327ab86/volumes"
Jul 28 17:41:49.731217 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:49.731183    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"]
Jul 28 17:41:49.731759 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:49.731427    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container" containerID="cri-o://100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168" gracePeriod=30
Jul 28 17:41:49.731759 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:49.731459    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kube-rbac-proxy" containerID="cri-o://fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3" gracePeriod=30
Jul 28 17:41:50.698799 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:50.698765    2563 generic.go:356] "Generic (PLEG): container finished" podID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerID="fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3" exitCode=2
Jul 28 17:41:50.698952 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:50.698838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" event={"ID":"23616cc5-959c-48e4-8651-b8fa86fecc0c","Type":"ContainerDied","Data":"fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3"}
Jul 28 17:41:51.974379 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:51.974343    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 28 17:41:53.879189 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.879167    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:41:53.940486 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.940459    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-rbac-proxy-sar-config\") pod \"23616cc5-959c-48e4-8651-b8fa86fecc0c\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") "
Jul 28 17:41:53.940607 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.940497    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23616cc5-959c-48e4-8651-b8fa86fecc0c-proxy-tls\") pod \"23616cc5-959c-48e4-8651-b8fa86fecc0c\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") "
Jul 28 17:41:53.940607 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.940516    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-cabundle-cert\") pod \"23616cc5-959c-48e4-8651-b8fa86fecc0c\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") "
Jul 28 17:41:53.940607 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.940561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23616cc5-959c-48e4-8651-b8fa86fecc0c-kserve-provision-location\") pod \"23616cc5-959c-48e4-8651-b8fa86fecc0c\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") "
Jul 28 17:41:53.940771 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.940622    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnhl4\" (UniqueName: \"kubernetes.io/projected/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-api-access-tnhl4\") pod \"23616cc5-959c-48e4-8651-b8fa86fecc0c\" (UID: \"23616cc5-959c-48e4-8651-b8fa86fecc0c\") "
Jul 28 17:41:53.940902 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.940869    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "23616cc5-959c-48e4-8651-b8fa86fecc0c" (UID: "23616cc5-959c-48e4-8651-b8fa86fecc0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:53.941002 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.940905    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "23616cc5-959c-48e4-8651-b8fa86fecc0c" (UID: "23616cc5-959c-48e4-8651-b8fa86fecc0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:41:53.941002 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.940908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23616cc5-959c-48e4-8651-b8fa86fecc0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "23616cc5-959c-48e4-8651-b8fa86fecc0c" (UID: "23616cc5-959c-48e4-8651-b8fa86fecc0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:41:53.942497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.942475    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/23616cc5-959c-48e4-8651-b8fa86fecc0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "23616cc5-959c-48e4-8651-b8fa86fecc0c" (UID: "23616cc5-959c-48e4-8651-b8fa86fecc0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:41:53.942608 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:53.942590    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-api-access-tnhl4" (OuterVolumeSpecName: "kube-api-access-tnhl4") pod "23616cc5-959c-48e4-8651-b8fa86fecc0c" (UID: "23616cc5-959c-48e4-8651-b8fa86fecc0c"). InnerVolumeSpecName "kube-api-access-tnhl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:41:54.041692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.041620    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnhl4\" (UniqueName: \"kubernetes.io/projected/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-api-access-tnhl4\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:41:54.041692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.041649    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:41:54.041692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.041660    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/23616cc5-959c-48e4-8651-b8fa86fecc0c-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:41:54.041692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.041686    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/23616cc5-959c-48e4-8651-b8fa86fecc0c-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:41:54.041692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.041693    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/23616cc5-959c-48e4-8651-b8fa86fecc0c-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:41:54.716633 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.716602    2563 generic.go:356] "Generic (PLEG): container finished" podID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerID="100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168" exitCode=0
Jul 28 17:41:54.716833 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.716656    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"
Jul 28 17:41:54.716833 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.716662    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" event={"ID":"23616cc5-959c-48e4-8651-b8fa86fecc0c","Type":"ContainerDied","Data":"100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168"}
Jul 28 17:41:54.716833 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.716739    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl" event={"ID":"23616cc5-959c-48e4-8651-b8fa86fecc0c","Type":"ContainerDied","Data":"9ca5bac9072a798f7f881680848ad8630321f3649e6ab189fbeba0367daaccff"}
Jul 28 17:41:54.716833 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.716776    2563 scope.go:117] "RemoveContainer" containerID="fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3"
Jul 28 17:41:54.725557 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.725538    2563 scope.go:117] "RemoveContainer" containerID="100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168"
Jul 28 17:41:54.732596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.732580    2563 scope.go:117] "RemoveContainer" containerID="ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825"
Jul 28 17:41:54.739265 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.739246    2563 scope.go:117] "RemoveContainer" containerID="fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3"
Jul 28 17:41:54.739524 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:41:54.739504    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3\": container with ID starting with fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3 not found: ID does not exist" containerID="fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3"
Jul 28 17:41:54.739611 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.739532    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3"} err="failed to get container status \"fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3\": rpc error: code = NotFound desc = could not find container \"fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3\": container with ID starting with fd9d4038f84bf4c7013a5b7bda2dfd39e31e1270d97da1d5554ed78c3d4cd1c3 not found: ID does not exist"
Jul 28 17:41:54.739611 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.739563    2563 scope.go:117] "RemoveContainer" containerID="100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168"
Jul 28 17:41:54.739886 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:41:54.739858    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168\": container with ID starting with 100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168 not found: ID does not exist" containerID="100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168"
Jul 28 17:41:54.739979 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.739892    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168"} err="failed to get container status \"100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168\": rpc error: code = NotFound desc = could not find container \"100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168\": container with ID starting with 100cca3d0651e063434fb99eb58301efe9ad1951595a849c6ca37b73ef90a168 not found: ID does not exist"
Jul 28 17:41:54.739979 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.739908    2563 scope.go:117] "RemoveContainer" containerID="ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825"
Jul 28 17:41:54.740263 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:41:54.740243    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825\": container with ID starting with ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825 not found: ID does not exist" containerID="ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825"
Jul 28 17:41:54.740340 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.740267    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825"} err="failed to get container status \"ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825\": rpc error: code = NotFound desc = could not find container \"ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825\": container with ID starting with ebe2c2c6fbad916fa52d39069aaa1a343fecb878b11c3f10a76ce59546f06825 not found: ID does not exist"
Jul 28 17:41:54.741922 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.741905    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"]
Jul 28 17:41:54.745106 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.745084    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-86766c4958-djmzl"]
Jul 28 17:41:54.886839 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.886813    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"]
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887090    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="storage-initializer"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887101    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="storage-initializer"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887111    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887117    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887137    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kube-rbac-proxy"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887144    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kube-rbac-proxy"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887150    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887156    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887166    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kube-rbac-proxy"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887171    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kube-rbac-proxy"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887191    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="storage-initializer"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887197    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="storage-initializer"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887245    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kserve-container"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887252    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kserve-container"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887259    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" containerName="kube-rbac-proxy"
Jul 28 17:41:54.887322 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.887267    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa9baed3-9ba9-4189-ad81-18e61327ab86" containerName="kube-rbac-proxy"
Jul 28 17:41:54.891375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.891353    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:54.893123 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.893102    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"]
Jul 28 17:41:54.894383 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.894362    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:41:54.894488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.894364    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:41:54.894488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.894413    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:41:54.895476 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.895459    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 28 17:41:54.895476 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.895466    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:41:54.895614 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.895515    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 28 17:41:54.895728 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.895715    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jwrfk\""
Jul 28 17:41:54.896302 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.896285    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:54.898700 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.898657    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-v2-predictor-serving-cert\""
Jul 28 17:41:54.903660 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.903637    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"]
Jul 28 17:41:54.907774 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.907753    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"]
Jul 28 17:41:54.947795 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.947768    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:54.947893 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.947805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mqw4r\" (UniqueName: \"kubernetes.io/projected/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-api-access-mqw4r\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:54.947893 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.947826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22a668-66f5-446a-864e-d793c3f5d43c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:54.947893 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.947843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:54.948020 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.947889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:54.948020 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.947931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:54.948020 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.947969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:54.948129 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.948044    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb9c6\" (UniqueName: \"kubernetes.io/projected/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-api-access-fb9c6\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:54.948129 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.948071    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:54.948129 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:54.948097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.048964 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.048910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb9c6\" (UniqueName: \"kubernetes.io/projected/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-api-access-fb9c6\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.048964 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.048935    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.048964 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.048960    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.049149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.048980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.049149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mqw4r\" (UniqueName: \"kubernetes.io/projected/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-api-access-mqw4r\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.049149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22a668-66f5-446a-864e-d793c3f5d43c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.049149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.049149 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:41:55.049081    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 17:41:55.049149 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.049149 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:41:55.049143    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls podName:bd73e697-88ea-4f12-92fb-e7c201b8a2dc nodeName:}" failed. No retries permitted until 2026-07-28 17:41:55.549123363 +0000 UTC m=+1492.482522775 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls") pod "isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" (UID: "bd73e697-88ea-4f12-92fb-e7c201b8a2dc") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 28 17:41:55.049505 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.049505 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049242    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.049505 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:41:55.049395    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-serving-cert: secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 17:41:55.049505 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049432    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22a668-66f5-446a-864e-d793c3f5d43c-kserve-provision-location\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.049505 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:41:55.049456    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls podName:bd22a668-66f5-446a-864e-d793c3f5d43c nodeName:}" failed. No retries permitted until 2026-07-28 17:41:55.549440065 +0000 UTC m=+1492.482839465 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls") pod "isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" (UID: "bd22a668-66f5-446a-864e-d793c3f5d43c") : secret "isvc-canary-rollback-v2-predictor-serving-cert" not found
Jul 28 17:41:55.049505 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.049735 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-cabundle-cert\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.049800 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.049891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049876    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.049926 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.049889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-cabundle-cert\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.057240 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.057203    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mqw4r\" (UniqueName: \"kubernetes.io/projected/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-api-access-mqw4r\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.057732 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.057712    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb9c6\" (UniqueName: \"kubernetes.io/projected/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-api-access-fb9c6\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.552630 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.552590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.552810 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.552704    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.555068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.555042    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls\") pod \"isvc-canary-rollback-v2-predictor-84f48b7656-wzk74\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.555197 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.555070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls\") pod \"isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.596795 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.596768    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23616cc5-959c-48e4-8651-b8fa86fecc0c" path="/var/lib/kubelet/pods/23616cc5-959c-48e4-8651-b8fa86fecc0c/volumes"
Jul 28 17:41:55.803650 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.803593    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:41:55.810641 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.810625    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:41:55.939491 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.939447    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"]
Jul 28 17:41:55.942010 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:41:55.941976    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbd73e697_88ea_4f12_92fb_e7c201b8a2dc.slice/crio-d5da044964c76e8deda9ea679346e05611d3a6009d0821118c7617e848604e4f WatchSource:0}: Error finding container d5da044964c76e8deda9ea679346e05611d3a6009d0821118c7617e848604e4f: Status 404 returned error can't find the container with id d5da044964c76e8deda9ea679346e05611d3a6009d0821118c7617e848604e4f
Jul 28 17:41:55.960490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:55.960003    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"]
Jul 28 17:41:56.728105 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:56.728068    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" event={"ID":"bd73e697-88ea-4f12-92fb-e7c201b8a2dc","Type":"ContainerStarted","Data":"904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347"}
Jul 28 17:41:56.728311 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:56.728135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" event={"ID":"bd73e697-88ea-4f12-92fb-e7c201b8a2dc","Type":"ContainerStarted","Data":"d5da044964c76e8deda9ea679346e05611d3a6009d0821118c7617e848604e4f"}
Jul 28 17:41:56.729597 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:56.729570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" event={"ID":"bd22a668-66f5-446a-864e-d793c3f5d43c","Type":"ContainerStarted","Data":"641748ba3e537608c4d484377e653dab13901f3487122dfec41d9f71c4cead47"}
Jul 28 17:41:56.729738 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:56.729631    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" event={"ID":"bd22a668-66f5-446a-864e-d793c3f5d43c","Type":"ContainerStarted","Data":"70ae224b64a834a2d3081ffd997be86a9080f4f6c684ca99f5ba4b0eca6c32a6"}
Jul 28 17:41:59.742948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:59.742921    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerID="904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347" exitCode=0
Jul 28 17:41:59.743331 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:59.742999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" event={"ID":"bd73e697-88ea-4f12-92fb-e7c201b8a2dc","Type":"ContainerDied","Data":"904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347"}
Jul 28 17:41:59.744586 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:59.744566    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerID="641748ba3e537608c4d484377e653dab13901f3487122dfec41d9f71c4cead47" exitCode=0
Jul 28 17:41:59.744699 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:41:59.744629    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" event={"ID":"bd22a668-66f5-446a-864e-d793c3f5d43c","Type":"ContainerDied","Data":"641748ba3e537608c4d484377e653dab13901f3487122dfec41d9f71c4cead47"}
Jul 28 17:42:00.751063 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.751022    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" event={"ID":"bd73e697-88ea-4f12-92fb-e7c201b8a2dc","Type":"ContainerStarted","Data":"23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8"}
Jul 28 17:42:00.751498 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.751120    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" event={"ID":"bd73e697-88ea-4f12-92fb-e7c201b8a2dc","Type":"ContainerStarted","Data":"1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937"}
Jul 28 17:42:00.751498 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.751324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:42:00.753151 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.753129    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" event={"ID":"bd22a668-66f5-446a-864e-d793c3f5d43c","Type":"ContainerStarted","Data":"0404ebe5c28dba14a4841e99b0683659ebb8889a571abbf3c4dd61929bd63250"}
Jul 28 17:42:00.753223 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.753177    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" event={"ID":"bd22a668-66f5-446a-864e-d793c3f5d43c","Type":"ContainerStarted","Data":"be7373fffd5b7f1eb6838c62ec7a6bc0d5970b730d9c5c354757690b19f202dc"}
Jul 28 17:42:00.753488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.753468    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:42:00.753597 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.753513    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:42:00.754771 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.754746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:42:00.772086 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.772047    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podStartSLOduration=3.101779714 podStartE2EDuration="6.772036682s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.670256968s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:00.770945341 +0000 UTC m=+1497.704344762" watchObservedRunningTime="2026-07-28 17:42:00.772036682 +0000 UTC m=+1497.705436101"
Jul 28 17:42:00.792930 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:00.792889    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podStartSLOduration=3.169985944 podStartE2EDuration="6.792877767s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.622891823s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:41:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:42:00.791636347 +0000 UTC m=+1497.725035767" watchObservedRunningTime="2026-07-28 17:42:00.792877767 +0000 UTC m=+1497.726277185"
Jul 28 17:42:01.757783 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:01.757734    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:42:01.758152 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:01.757954    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:42:01.759101 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:01.759078    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:42:02.761954 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:02.761914    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:42:06.761707 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:06.761659    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:42:06.762201 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:06.762172    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:42:07.766817 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:07.766789    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:42:07.767324 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:07.767298    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:42:16.762412 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:16.762371    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:42:17.767613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:17.767574    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:42:26.762829 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:26.762750    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:42:27.767440 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:27.767402    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:42:36.762393 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:36.762350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:42:37.767261 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:37.767215    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:42:46.762428 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:46.762389    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:42:47.767445 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:47.767406    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:42:56.762787 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:56.762746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:42:57.767346 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:42:57.767304    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:43:06.762892 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:06.762854    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 28 17:43:07.768188 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:07.768153    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:43:11.596215 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:11.596188    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:43:16.763012 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:16.762983    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:43:19.894520 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:19.894484    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"]
Jul 28 17:43:19.894940 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:19.894705    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container" containerID="cri-o://be7373fffd5b7f1eb6838c62ec7a6bc0d5970b730d9c5c354757690b19f202dc" gracePeriod=30
Jul 28 17:43:19.894940 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:19.894738    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kube-rbac-proxy" containerID="cri-o://0404ebe5c28dba14a4841e99b0683659ebb8889a571abbf3c4dd61929bd63250" gracePeriod=30
Jul 28 17:43:20.065618 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:20.065588    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerID="0404ebe5c28dba14a4841e99b0683659ebb8889a571abbf3c4dd61929bd63250" exitCode=2
Jul 28 17:43:20.065775 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:20.065681    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" event={"ID":"bd22a668-66f5-446a-864e-d793c3f5d43c","Type":"ContainerDied","Data":"0404ebe5c28dba14a4841e99b0683659ebb8889a571abbf3c4dd61929bd63250"}
Jul 28 17:43:21.758488 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:21.758451    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 28 17:43:24.086172 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.086139    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerID="be7373fffd5b7f1eb6838c62ec7a6bc0d5970b730d9c5c354757690b19f202dc" exitCode=0
Jul 28 17:43:24.086504 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.086206    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" event={"ID":"bd22a668-66f5-446a-864e-d793c3f5d43c","Type":"ContainerDied","Data":"be7373fffd5b7f1eb6838c62ec7a6bc0d5970b730d9c5c354757690b19f202dc"}
Jul 28 17:43:24.134655 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.134630    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:43:24.288345 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-rbac-proxy-sar-config\") pod \"bd22a668-66f5-446a-864e-d793c3f5d43c\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") "
Jul 28 17:43:24.288345 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288341    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mqw4r\" (UniqueName: \"kubernetes.io/projected/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-api-access-mqw4r\") pod \"bd22a668-66f5-446a-864e-d793c3f5d43c\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") "
Jul 28 17:43:24.288497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls\") pod \"bd22a668-66f5-446a-864e-d793c3f5d43c\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") "
Jul 28 17:43:24.288497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22a668-66f5-446a-864e-d793c3f5d43c-kserve-provision-location\") pod \"bd22a668-66f5-446a-864e-d793c3f5d43c\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") "
Jul 28 17:43:24.288609 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288520    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-cabundle-cert\") pod \"bd22a668-66f5-446a-864e-d793c3f5d43c\" (UID: \"bd22a668-66f5-446a-864e-d793c3f5d43c\") "
Jul 28 17:43:24.288692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288610    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd22a668-66f5-446a-864e-d793c3f5d43c" (UID: "bd22a668-66f5-446a-864e-d793c3f5d43c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:24.288803 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288781    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:43:24.288933 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288812    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd22a668-66f5-446a-864e-d793c3f5d43c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd22a668-66f5-446a-864e-d793c3f5d43c" (UID: "bd22a668-66f5-446a-864e-d793c3f5d43c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:43:24.288933 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.288890    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd22a668-66f5-446a-864e-d793c3f5d43c" (UID: "bd22a668-66f5-446a-864e-d793c3f5d43c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:43:24.290319 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.290298    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-api-access-mqw4r" (OuterVolumeSpecName: "kube-api-access-mqw4r") pod "bd22a668-66f5-446a-864e-d793c3f5d43c" (UID: "bd22a668-66f5-446a-864e-d793c3f5d43c"). InnerVolumeSpecName "kube-api-access-mqw4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:43:24.290438 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.290423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd22a668-66f5-446a-864e-d793c3f5d43c" (UID: "bd22a668-66f5-446a-864e-d793c3f5d43c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:43:24.390125 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.390101    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mqw4r\" (UniqueName: \"kubernetes.io/projected/bd22a668-66f5-446a-864e-d793c3f5d43c-kube-api-access-mqw4r\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:43:24.390125 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.390122    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd22a668-66f5-446a-864e-d793c3f5d43c-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:43:24.390274 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.390132    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd22a668-66f5-446a-864e-d793c3f5d43c-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:43:24.390274 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:24.390141    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd22a668-66f5-446a-864e-d793c3f5d43c-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:43:25.092297 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:25.092264    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"
Jul 28 17:43:25.092297 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:25.092263    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74" event={"ID":"bd22a668-66f5-446a-864e-d793c3f5d43c","Type":"ContainerDied","Data":"70ae224b64a834a2d3081ffd997be86a9080f4f6c684ca99f5ba4b0eca6c32a6"}
Jul 28 17:43:25.092867 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:25.092334    2563 scope.go:117] "RemoveContainer" containerID="0404ebe5c28dba14a4841e99b0683659ebb8889a571abbf3c4dd61929bd63250"
Jul 28 17:43:25.100573 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:25.100557    2563 scope.go:117] "RemoveContainer" containerID="be7373fffd5b7f1eb6838c62ec7a6bc0d5970b730d9c5c354757690b19f202dc"
Jul 28 17:43:25.107733 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:25.107717    2563 scope.go:117] "RemoveContainer" containerID="641748ba3e537608c4d484377e653dab13901f3487122dfec41d9f71c4cead47"
Jul 28 17:43:25.113860 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:25.113838    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"]
Jul 28 17:43:25.119045 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:25.119027    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-v2-predictor-84f48b7656-wzk74"]
Jul 28 17:43:25.596542 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:43:25.596503    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" path="/var/lib/kubelet/pods/bd22a668-66f5-446a-864e-d793c3f5d43c/volumes"
Jul 28 17:44:39.997985 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:39.997947    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"]
Jul 28 17:44:39.998535 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:39.998233    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" containerID="cri-o://1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937" gracePeriod=30
Jul 28 17:44:39.998535 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:39.998310    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kube-rbac-proxy" containerID="cri-o://23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8" gracePeriod=30
Jul 28 17:44:40.378074 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:40.378044    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerID="23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8" exitCode=2
Jul 28 17:44:40.378074 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:40.378077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" event={"ID":"bd73e697-88ea-4f12-92fb-e7c201b8a2dc","Type":"ContainerDied","Data":"23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8"}
Jul 28 17:44:41.591997 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:41.591949    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 28 17:44:42.763043 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:42.763004    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 28 17:44:44.245542 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.245520    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:44:44.289196 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.289169    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-rbac-proxy-sar-config\") pod \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") "
Jul 28 17:44:44.289311 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.289212    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fb9c6\" (UniqueName: \"kubernetes.io/projected/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-api-access-fb9c6\") pod \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") "
Jul 28 17:44:44.289311 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.289261    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kserve-provision-location\") pod \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") "
Jul 28 17:44:44.289311 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.289290    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-cabundle-cert\") pod \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") "
Jul 28 17:44:44.289444 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.289336    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls\") pod \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\" (UID: \"bd73e697-88ea-4f12-92fb-e7c201b8a2dc\") "
Jul 28 17:44:44.289613 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.289589    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "bd73e697-88ea-4f12-92fb-e7c201b8a2dc" (UID: "bd73e697-88ea-4f12-92fb-e7c201b8a2dc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:44:44.289658 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.289621    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "bd73e697-88ea-4f12-92fb-e7c201b8a2dc" (UID: "bd73e697-88ea-4f12-92fb-e7c201b8a2dc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:44:44.289658 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.289628    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "bd73e697-88ea-4f12-92fb-e7c201b8a2dc" (UID: "bd73e697-88ea-4f12-92fb-e7c201b8a2dc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:44:44.291278 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.291251    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "bd73e697-88ea-4f12-92fb-e7c201b8a2dc" (UID: "bd73e697-88ea-4f12-92fb-e7c201b8a2dc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:44:44.291394 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.291379    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-api-access-fb9c6" (OuterVolumeSpecName: "kube-api-access-fb9c6") pod "bd73e697-88ea-4f12-92fb-e7c201b8a2dc" (UID: "bd73e697-88ea-4f12-92fb-e7c201b8a2dc"). InnerVolumeSpecName "kube-api-access-fb9c6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:44:44.390891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.390841    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:44:44.390891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.390860    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:44:44.390891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.390871    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:44:44.390891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.390879    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:44:44.390891 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.390888    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb9c6\" (UniqueName: \"kubernetes.io/projected/bd73e697-88ea-4f12-92fb-e7c201b8a2dc-kube-api-access-fb9c6\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:44:44.395532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.395506    2563 generic.go:356] "Generic (PLEG): container finished" podID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerID="1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937" exitCode=0
Jul 28 17:44:44.395623 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.395566    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"
Jul 28 17:44:44.395623 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.395586    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" event={"ID":"bd73e697-88ea-4f12-92fb-e7c201b8a2dc","Type":"ContainerDied","Data":"1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937"}
Jul 28 17:44:44.395727 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.395646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk" event={"ID":"bd73e697-88ea-4f12-92fb-e7c201b8a2dc","Type":"ContainerDied","Data":"d5da044964c76e8deda9ea679346e05611d3a6009d0821118c7617e848604e4f"}
Jul 28 17:44:44.395727 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.395691    2563 scope.go:117] "RemoveContainer" containerID="23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8"
Jul 28 17:44:44.403726 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.403704    2563 scope.go:117] "RemoveContainer" containerID="1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937"
Jul 28 17:44:44.410595 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.410578    2563 scope.go:117] "RemoveContainer" containerID="904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347"
Jul 28 17:44:44.416687 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.416657    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"]
Jul 28 17:44:44.417585 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.417568    2563 scope.go:117] "RemoveContainer" containerID="23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8"
Jul 28 17:44:44.417894 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:44:44.417877    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8\": container with ID starting with 23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8 not found: ID does not exist" containerID="23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8"
Jul 28 17:44:44.417969 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.417901    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8"} err="failed to get container status \"23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8\": rpc error: code = NotFound desc = could not find container \"23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8\": container with ID starting with 23391bb9839a72233213c5cb0d42b4de1f3e21648f0c4948b182d037afb8aba8 not found: ID does not exist"
Jul 28 17:44:44.417969 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.417917    2563 scope.go:117] "RemoveContainer" containerID="1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937"
Jul 28 17:44:44.418143 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:44:44.418124    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937\": container with ID starting with 1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937 not found: ID does not exist" containerID="1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937"
Jul 28 17:44:44.418202 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.418151    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937"} err="failed to get container status \"1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937\": rpc error: code = NotFound desc = could not find container \"1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937\": container with ID starting with 1f1121ddfce3650dcfa383c00c201538e6dd9637c50dcb396b83c340b2488937 not found: ID does not exist"
Jul 28 17:44:44.418202 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.418166    2563 scope.go:117] "RemoveContainer" containerID="904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347"
Jul 28 17:44:44.418482 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:44:44.418457    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347\": container with ID starting with 904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347 not found: ID does not exist" containerID="904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347"
Jul 28 17:44:44.418635 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.418490    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347"} err="failed to get container status \"904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347\": rpc error: code = NotFound desc = could not find container \"904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347\": container with ID starting with 904ebcdc28bda08083640c16b0a05c008271c3313ef4246a1d481d7c4ce08347 not found: ID does not exist"
Jul 28 17:44:44.420216 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:44.420197    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-5bbb8c8cfc-t6ktk"]
Jul 28 17:44:45.172665 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.172634    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"]
Jul 28 17:44:45.172961 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.172949    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container"
Jul 28 17:44:45.173009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.172963    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container"
Jul 28 17:44:45.173009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.172972    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="storage-initializer"
Jul 28 17:44:45.173009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.172977    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="storage-initializer"
Jul 28 17:44:45.173009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.172984    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container"
Jul 28 17:44:45.173009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.172990    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container"
Jul 28 17:44:45.173009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173003    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kube-rbac-proxy"
Jul 28 17:44:45.173009 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173008    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kube-rbac-proxy"
Jul 28 17:44:45.173221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173015    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="storage-initializer"
Jul 28 17:44:45.173221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173020    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="storage-initializer"
Jul 28 17:44:45.173221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173026    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kube-rbac-proxy"
Jul 28 17:44:45.173221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173030    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kube-rbac-proxy"
Jul 28 17:44:45.173221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173089    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kube-rbac-proxy"
Jul 28 17:44:45.173221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173097    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kube-rbac-proxy"
Jul 28 17:44:45.173221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173104    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd22a668-66f5-446a-864e-d793c3f5d43c" containerName="kserve-container"
Jul 28 17:44:45.173221 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.173110    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" containerName="kserve-container"
Jul 28 17:44:45.177716 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.177697    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.180685 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.180631    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jwrfk\""
Jul 28 17:44:45.180900 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.180885    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:44:45.180987 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.180896    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:44:45.181842 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.181825    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 28 17:44:45.181950 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.181874    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:44:45.181950 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.181936    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:44:45.182065 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.182022    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 28 17:44:45.186565 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.186544    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"]
Jul 28 17:44:45.195622 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.195599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.195732 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.195644    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.195732 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.195664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.195858 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.195761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e43bd28-72c2-4dfb-9280-5c1384265d09-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.195858 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.195818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgcjv\" (UniqueName: \"kubernetes.io/projected/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-api-access-qgcjv\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.296142 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.296114    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.296142 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.296144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.296546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.296170    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e43bd28-72c2-4dfb-9280-5c1384265d09-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.296546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.296208    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgcjv\" (UniqueName: \"kubernetes.io/projected/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-api-access-qgcjv\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.296546 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.296248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.296546 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:44:45.296266    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 17:44:45.296546 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:44:45.296335    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls podName:8e43bd28-72c2-4dfb-9280-5c1384265d09 nodeName:}" failed. No retries permitted until 2026-07-28 17:44:45.796317788 +0000 UTC m=+1662.729717185 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls") pod "isvc-canary-stop-predictor-86dc9fdb65-spv99" (UID: "8e43bd28-72c2-4dfb-9280-5c1384265d09") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 28 17:44:45.296743 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.296652    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e43bd28-72c2-4dfb-9280-5c1384265d09-kserve-provision-location\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.296992 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.296972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-cabundle-cert\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.297029 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.296972    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.306290 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.306271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgcjv\" (UniqueName: \"kubernetes.io/projected/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-api-access-qgcjv\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.596220 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.596189    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bd73e697-88ea-4f12-92fb-e7c201b8a2dc" path="/var/lib/kubelet/pods/bd73e697-88ea-4f12-92fb-e7c201b8a2dc/volumes"
Jul 28 17:44:45.800668 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.800643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:45.802956 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:45.802930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls\") pod \"isvc-canary-stop-predictor-86dc9fdb65-spv99\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:46.088316 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:46.088292    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:46.214732 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:46.214703    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"]
Jul 28 17:44:46.217908 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:44:46.217871    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e43bd28_72c2_4dfb_9280_5c1384265d09.slice/crio-a344185b62914b7fa707d9b26a819cdb4d962a398272c32e07a709626354d5f0 WatchSource:0}: Error finding container a344185b62914b7fa707d9b26a819cdb4d962a398272c32e07a709626354d5f0: Status 404 returned error can't find the container with id a344185b62914b7fa707d9b26a819cdb4d962a398272c32e07a709626354d5f0
Jul 28 17:44:46.408864 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:46.408789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" event={"ID":"8e43bd28-72c2-4dfb-9280-5c1384265d09","Type":"ContainerStarted","Data":"9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e"}
Jul 28 17:44:46.409185 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:46.408863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" event={"ID":"8e43bd28-72c2-4dfb-9280-5c1384265d09","Type":"ContainerStarted","Data":"a344185b62914b7fa707d9b26a819cdb4d962a398272c32e07a709626354d5f0"}
Jul 28 17:44:50.424680 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:50.424647    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerID="9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e" exitCode=0
Jul 28 17:44:50.425035 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:50.424706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" event={"ID":"8e43bd28-72c2-4dfb-9280-5c1384265d09","Type":"ContainerDied","Data":"9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e"}
Jul 28 17:44:51.430452 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:51.430419    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" event={"ID":"8e43bd28-72c2-4dfb-9280-5c1384265d09","Type":"ContainerStarted","Data":"43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517"}
Jul 28 17:44:51.430854 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:51.430503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" event={"ID":"8e43bd28-72c2-4dfb-9280-5c1384265d09","Type":"ContainerStarted","Data":"d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579"}
Jul 28 17:44:51.430854 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:51.430781    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:51.451357 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:51.451314    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podStartSLOduration=2.92776742 podStartE2EDuration="6.451300855s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.523533435s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:44:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:44:51.449759963 +0000 UTC m=+1668.383159372" watchObservedRunningTime="2026-07-28 17:44:51.451300855 +0000 UTC m=+1668.384700342"
Jul 28 17:44:52.435358 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:52.435323    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:52.436565 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:52.436540    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:44:53.438693 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:53.438638    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:44:58.442688 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:58.442643    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:44:58.443227 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:44:58.443197    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:45:08.443165 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:45:08.443123    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:45:18.443846 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:45:18.443805    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:45:28.443964 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:45:28.443869    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:45:38.443798 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:45:38.443758    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:45:48.443799 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:45:48.443759    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:45:58.443477 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:45:58.443437    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:46:08.443694 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:08.443639    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:46:15.205575 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:15.205542    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"]
Jul 28 17:46:15.206150 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:15.205858    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" containerID="cri-o://d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579" gracePeriod=30
Jul 28 17:46:15.206150 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:15.205930    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kube-rbac-proxy" containerID="cri-o://43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517" gracePeriod=30
Jul 28 17:46:15.763041 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:15.763004    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerID="43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517" exitCode=2
Jul 28 17:46:15.763213 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:15.763074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" event={"ID":"8e43bd28-72c2-4dfb-9280-5c1384265d09","Type":"ContainerDied","Data":"43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517"}
Jul 28 17:46:18.439706 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:18.439646    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 28 17:46:18.443311 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:18.443287    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 28 17:46:19.155226 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.155203    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:46:19.207387 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207370    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls\") pod \"8e43bd28-72c2-4dfb-9280-5c1384265d09\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") "
Jul 28 17:46:19.207505 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207399    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e43bd28-72c2-4dfb-9280-5c1384265d09-kserve-provision-location\") pod \"8e43bd28-72c2-4dfb-9280-5c1384265d09\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") "
Jul 28 17:46:19.207505 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207436    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-rbac-proxy-sar-config\") pod \"8e43bd28-72c2-4dfb-9280-5c1384265d09\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") "
Jul 28 17:46:19.207505 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207490    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qgcjv\" (UniqueName: \"kubernetes.io/projected/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-api-access-qgcjv\") pod \"8e43bd28-72c2-4dfb-9280-5c1384265d09\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") "
Jul 28 17:46:19.207759 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207543    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-cabundle-cert\") pod \"8e43bd28-72c2-4dfb-9280-5c1384265d09\" (UID: \"8e43bd28-72c2-4dfb-9280-5c1384265d09\") "
Jul 28 17:46:19.207759 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207655    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e43bd28-72c2-4dfb-9280-5c1384265d09-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8e43bd28-72c2-4dfb-9280-5c1384265d09" (UID: "8e43bd28-72c2-4dfb-9280-5c1384265d09"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:46:19.207881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207794    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8e43bd28-72c2-4dfb-9280-5c1384265d09-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:46:19.207881 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207867    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8e43bd28-72c2-4dfb-9280-5c1384265d09" (UID: "8e43bd28-72c2-4dfb-9280-5c1384265d09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:19.207959 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.207945    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8e43bd28-72c2-4dfb-9280-5c1384265d09" (UID: "8e43bd28-72c2-4dfb-9280-5c1384265d09"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:46:19.209384 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.209352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8e43bd28-72c2-4dfb-9280-5c1384265d09" (UID: "8e43bd28-72c2-4dfb-9280-5c1384265d09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:46:19.209461 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.209443    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-api-access-qgcjv" (OuterVolumeSpecName: "kube-api-access-qgcjv") pod "8e43bd28-72c2-4dfb-9280-5c1384265d09" (UID: "8e43bd28-72c2-4dfb-9280-5c1384265d09"). InnerVolumeSpecName "kube-api-access-qgcjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:46:19.308953 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.308904    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:46:19.308953 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.308924    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8e43bd28-72c2-4dfb-9280-5c1384265d09-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:46:19.308953 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.308934    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:46:19.308953 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.308945    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgcjv\" (UniqueName: \"kubernetes.io/projected/8e43bd28-72c2-4dfb-9280-5c1384265d09-kube-api-access-qgcjv\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:46:19.779935 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.779902    2563 generic.go:356] "Generic (PLEG): container finished" podID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerID="d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579" exitCode=0
Jul 28 17:46:19.780282 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.779943    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" event={"ID":"8e43bd28-72c2-4dfb-9280-5c1384265d09","Type":"ContainerDied","Data":"d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579"}
Jul 28 17:46:19.780282 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.779958    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"
Jul 28 17:46:19.780282 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.779995    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99" event={"ID":"8e43bd28-72c2-4dfb-9280-5c1384265d09","Type":"ContainerDied","Data":"a344185b62914b7fa707d9b26a819cdb4d962a398272c32e07a709626354d5f0"}
Jul 28 17:46:19.780282 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.780024    2563 scope.go:117] "RemoveContainer" containerID="43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517"
Jul 28 17:46:19.788518 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.788493    2563 scope.go:117] "RemoveContainer" containerID="d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579"
Jul 28 17:46:19.795387 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.795371    2563 scope.go:117] "RemoveContainer" containerID="9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e"
Jul 28 17:46:19.800160 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.800141    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"]
Jul 28 17:46:19.802653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.802628    2563 scope.go:117] "RemoveContainer" containerID="43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517"
Jul 28 17:46:19.802909 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:46:19.802892    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517\": container with ID starting with 43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517 not found: ID does not exist" containerID="43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517"
Jul 28 17:46:19.802974 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.802917    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517"} err="failed to get container status \"43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517\": rpc error: code = NotFound desc = could not find container \"43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517\": container with ID starting with 43f33015822f201785b07235a8b6e17c6b56e08627f17212433c8e354d237517 not found: ID does not exist"
Jul 28 17:46:19.802974 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.802934    2563 scope.go:117] "RemoveContainer" containerID="d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579"
Jul 28 17:46:19.803173 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:46:19.803153    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579\": container with ID starting with d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579 not found: ID does not exist" containerID="d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579"
Jul 28 17:46:19.803216 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.803181    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579"} err="failed to get container status \"d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579\": rpc error: code = NotFound desc = could not find container \"d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579\": container with ID starting with d2414fce81e6533697956b16e65efe9624ce47e7002e9d2f1e554e3d79ee7579 not found: ID does not exist"
Jul 28 17:46:19.803216 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.803198    2563 scope.go:117] "RemoveContainer" containerID="9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e"
Jul 28 17:46:19.803486 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:46:19.803459    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e\": container with ID starting with 9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e not found: ID does not exist" containerID="9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e"
Jul 28 17:46:19.803486 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.803493    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e"} err="failed to get container status \"9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e\": rpc error: code = NotFound desc = could not find container \"9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e\": container with ID starting with 9a47b52becec43f089c6569a509354f747d0bad65a05a369997f07a44693911e not found: ID does not exist"
Jul 28 17:46:19.807016 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:19.806997    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-86dc9fdb65-spv99"]
Jul 28 17:46:21.596543 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:21.596509    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" path="/var/lib/kubelet/pods/8e43bd28-72c2-4dfb-9280-5c1384265d09/volumes"
Jul 28 17:46:25.507668 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.507633    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"]
Jul 28 17:46:25.508068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.507932    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kube-rbac-proxy"
Jul 28 17:46:25.508068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.507943    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kube-rbac-proxy"
Jul 28 17:46:25.508068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.507965    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="storage-initializer"
Jul 28 17:46:25.508068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.507970    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="storage-initializer"
Jul 28 17:46:25.508068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.507976    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container"
Jul 28 17:46:25.508068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.507981    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container"
Jul 28 17:46:25.508068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.508024    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kube-rbac-proxy"
Jul 28 17:46:25.508068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.508031    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8e43bd28-72c2-4dfb-9280-5c1384265d09" containerName="kserve-container"
Jul 28 17:46:25.513743 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.513701    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.517033 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.517004    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:46:25.517172 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.517046    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:46:25.517172 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.517046    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ede845-kube-rbac-proxy-sar-config\""
Jul 28 17:46:25.517172 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.517093    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:46:25.517172 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.517161    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-primary-ede845-predictor-serving-cert\""
Jul 28 17:46:25.517375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.517187    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:46:25.517375 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.517208    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jwrfk\""
Jul 28 17:46:25.522854 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.522828    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"]
Jul 28 17:46:25.665367 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.665337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90c8ef-65b3-4463-ac92-8628a1d506d3-proxy-tls\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.665367 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.665366    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kserve-provision-location\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.665608 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.665387    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.665608 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.665425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-cabundle-cert\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.665608 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.665471    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4lb5\" (UniqueName: \"kubernetes.io/projected/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-api-access-c4lb5\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.766289 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.766198    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90c8ef-65b3-4463-ac92-8628a1d506d3-proxy-tls\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.766289 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.766238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kserve-provision-location\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.766289 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.766263    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.766562 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.766308    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-cabundle-cert\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.766562 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.766333    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4lb5\" (UniqueName: \"kubernetes.io/projected/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-api-access-c4lb5\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.766692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.766626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kserve-provision-location\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.767061 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.767036    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-cabundle-cert\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.767114 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.767099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-rbac-proxy-sar-config\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.768712 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.768689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90c8ef-65b3-4463-ac92-8628a1d506d3-proxy-tls\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.775249 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.775228    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4lb5\" (UniqueName: \"kubernetes.io/projected/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-api-access-c4lb5\") pod \"isvc-primary-ede845-predictor-577854c8df-5cwfn\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") " pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.825079 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.825056    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:25.951273 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:25.948898    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"]
Jul 28 17:46:25.952129 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:46:25.952101    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c90c8ef_65b3_4463_ac92_8628a1d506d3.slice/crio-d345540926577763f59cc219a6ee0c2dc9b0d3a31eae97760961a7a1d7b7d42e WatchSource:0}: Error finding container d345540926577763f59cc219a6ee0c2dc9b0d3a31eae97760961a7a1d7b7d42e: Status 404 returned error can't find the container with id d345540926577763f59cc219a6ee0c2dc9b0d3a31eae97760961a7a1d7b7d42e
Jul 28 17:46:26.808336 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:26.808305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" event={"ID":"6c90c8ef-65b3-4463-ac92-8628a1d506d3","Type":"ContainerStarted","Data":"fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95"}
Jul 28 17:46:26.808829 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:26.808357    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" event={"ID":"6c90c8ef-65b3-4463-ac92-8628a1d506d3","Type":"ContainerStarted","Data":"d345540926577763f59cc219a6ee0c2dc9b0d3a31eae97760961a7a1d7b7d42e"}
Jul 28 17:46:29.820911 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:29.820875    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerID="fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95" exitCode=0
Jul 28 17:46:29.821369 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:29.820913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" event={"ID":"6c90c8ef-65b3-4463-ac92-8628a1d506d3","Type":"ContainerDied","Data":"fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95"}
Jul 28 17:46:30.827278 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:30.827249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" event={"ID":"6c90c8ef-65b3-4463-ac92-8628a1d506d3","Type":"ContainerStarted","Data":"46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4"}
Jul 28 17:46:30.827639 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:30.827330    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" event={"ID":"6c90c8ef-65b3-4463-ac92-8628a1d506d3","Type":"ContainerStarted","Data":"42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5"}
Jul 28 17:46:30.827639 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:30.827511    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:30.846872 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:30.846833    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podStartSLOduration=2.275624802 podStartE2EDuration="5.846821435s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.571196633s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:46:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:46:30.8462212 +0000 UTC m=+1767.779620618" watchObservedRunningTime="2026-07-28 17:46:30.846821435 +0000 UTC m=+1767.780220853"
Jul 28 17:46:31.831109 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:31.831079    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:31.832252 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:31.832224    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:46:32.835127 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:32.835082    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:46:37.839185 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:37.839153    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:46:37.839650 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:37.839619    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:46:47.840449 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:47.840410    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:46:57.839719 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:46:57.839622    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:47:07.839942 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:47:07.839898    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:47:17.840298 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:47:17.840254    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:47:27.840281 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:47:27.840242    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:47:37.839937 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:47:37.839896    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:47:47.840864 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:47:47.840833    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:48:10.712137 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.712103    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"]
Jul 28 17:48:10.715390 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.715375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.718093 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.718062    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ede845-predictor-serving-cert\""
Jul 28 17:48:10.718233 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.718156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-secret-ede845\""
Jul 28 17:48:10.718233 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.718164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"invalid-s3-sa-ede845-dockercfg-xtqxf\""
Jul 28 17:48:10.718233 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.718186    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-secondary-ede845-kube-rbac-proxy-sar-config\""
Jul 28 17:48:10.725394 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.725374    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"]
Jul 28 17:48:10.830918 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.830878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqpfc\" (UniqueName: \"kubernetes.io/projected/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-api-access-nqpfc\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.831093 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.830930    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-cabundle-cert\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.831093 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.830973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a51eb9-3bfe-42e3-b353-afcd44834db0-kserve-provision-location\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.831093 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.831027    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.831222 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.831098    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a51eb9-3bfe-42e3-b353-afcd44834db0-proxy-tls\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.931967 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.931938    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqpfc\" (UniqueName: \"kubernetes.io/projected/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-api-access-nqpfc\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.932117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.931974    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-cabundle-cert\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.932117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.932014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a51eb9-3bfe-42e3-b353-afcd44834db0-kserve-provision-location\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.932117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.932042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.932117 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.932103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a51eb9-3bfe-42e3-b353-afcd44834db0-proxy-tls\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.932473 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.932451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a51eb9-3bfe-42e3-b353-afcd44834db0-kserve-provision-location\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.932781 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.932764    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-rbac-proxy-sar-config\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.932835 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.932768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-cabundle-cert\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.934598 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.934574    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a51eb9-3bfe-42e3-b353-afcd44834db0-proxy-tls\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:10.940159 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:10.940136    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqpfc\" (UniqueName: \"kubernetes.io/projected/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-api-access-nqpfc\") pod \"isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") " pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:11.026323 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:11.026252    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:11.154023 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:11.153994    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"]
Jul 28 17:48:11.156200 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:48:11.156167    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a51eb9_3bfe_42e3_b353_afcd44834db0.slice/crio-1a0a144ee55d94850a7951891b5aa42d5e62b546e6190922fea672cbc04984a5 WatchSource:0}: Error finding container 1a0a144ee55d94850a7951891b5aa42d5e62b546e6190922fea672cbc04984a5: Status 404 returned error can't find the container with id 1a0a144ee55d94850a7951891b5aa42d5e62b546e6190922fea672cbc04984a5
Jul 28 17:48:11.226849 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:11.226819    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv" event={"ID":"02a51eb9-3bfe-42e3-b353-afcd44834db0","Type":"ContainerStarted","Data":"9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2"}
Jul 28 17:48:11.226948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:11.226875    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv" event={"ID":"02a51eb9-3bfe-42e3-b353-afcd44834db0","Type":"ContainerStarted","Data":"1a0a144ee55d94850a7951891b5aa42d5e62b546e6190922fea672cbc04984a5"}
Jul 28 17:48:15.243039 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:15.243012    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_02a51eb9-3bfe-42e3-b353-afcd44834db0/storage-initializer/0.log"
Jul 28 17:48:15.243407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:15.243047    2563 generic.go:356] "Generic (PLEG): container finished" podID="02a51eb9-3bfe-42e3-b353-afcd44834db0" containerID="9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2" exitCode=1
Jul 28 17:48:15.243407 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:15.243126    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv" event={"ID":"02a51eb9-3bfe-42e3-b353-afcd44834db0","Type":"ContainerDied","Data":"9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2"}
Jul 28 17:48:16.248344 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:16.248315    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_02a51eb9-3bfe-42e3-b353-afcd44834db0/storage-initializer/0.log"
Jul 28 17:48:16.248747 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:16.248373    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv" event={"ID":"02a51eb9-3bfe-42e3-b353-afcd44834db0","Type":"ContainerStarted","Data":"2f59ca3cb4711c83e900d64539a866635ba74f972fdbe91b3dabf051d538e27e"}
Jul 28 17:48:18.257599 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:18.257569    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_02a51eb9-3bfe-42e3-b353-afcd44834db0/storage-initializer/1.log"
Jul 28 17:48:18.258059 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:18.257995    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_02a51eb9-3bfe-42e3-b353-afcd44834db0/storage-initializer/0.log"
Jul 28 17:48:18.258059 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:18.258035    2563 generic.go:356] "Generic (PLEG): container finished" podID="02a51eb9-3bfe-42e3-b353-afcd44834db0" containerID="2f59ca3cb4711c83e900d64539a866635ba74f972fdbe91b3dabf051d538e27e" exitCode=1
Jul 28 17:48:18.258163 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:18.258102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv" event={"ID":"02a51eb9-3bfe-42e3-b353-afcd44834db0","Type":"ContainerDied","Data":"2f59ca3cb4711c83e900d64539a866635ba74f972fdbe91b3dabf051d538e27e"}
Jul 28 17:48:18.258207 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:18.258167    2563 scope.go:117] "RemoveContainer" containerID="9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2"
Jul 28 17:48:18.258550 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:18.258517    2563 scope.go:117] "RemoveContainer" containerID="9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2"
Jul 28 17:48:18.270446 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:18.270424    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_kserve-ci-e2e-test(02a51eb9-3bfe-42e3-b353-afcd44834db0)\"" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv" podUID="02a51eb9-3bfe-42e3-b353-afcd44834db0"
Jul 28 17:48:18.271024 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:18.270997    2563 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_kserve-ci-e2e-test_02a51eb9-3bfe-42e3-b353-afcd44834db0_0 in pod sandbox 1a0a144ee55d94850a7951891b5aa42d5e62b546e6190922fea672cbc04984a5 from index: no such id: '9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2'" containerID="9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2"
Jul 28 17:48:18.271116 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:18.271036    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_kserve-ci-e2e-test_02a51eb9-3bfe-42e3-b353-afcd44834db0_0 in pod sandbox 1a0a144ee55d94850a7951891b5aa42d5e62b546e6190922fea672cbc04984a5 from index: no such id: '9b9d94c0055f615da1cf5367f46f9a99cb4a7eb7313d4bdfbbe55074d58389d2'"
Jul 28 17:48:19.263429 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:19.263404    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_02a51eb9-3bfe-42e3-b353-afcd44834db0/storage-initializer/1.log"
Jul 28 17:48:26.702222 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.702138    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"]
Jul 28 17:48:26.702657 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.702408    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" containerID="cri-o://42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5" gracePeriod=30
Jul 28 17:48:26.702657 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.702440    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kube-rbac-proxy" containerID="cri-o://46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4" gracePeriod=30
Jul 28 17:48:26.756771 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.756742    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"]
Jul 28 17:48:26.878799 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.878770    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_02a51eb9-3bfe-42e3-b353-afcd44834db0/storage-initializer/1.log"
Jul 28 17:48:26.881222 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.881204    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:26.950896 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.950868    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-rbac-proxy-sar-config\") pod \"02a51eb9-3bfe-42e3-b353-afcd44834db0\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") "
Jul 28 17:48:26.950896 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.950897    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a51eb9-3bfe-42e3-b353-afcd44834db0-proxy-tls\") pod \"02a51eb9-3bfe-42e3-b353-afcd44834db0\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") "
Jul 28 17:48:26.951089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.950922    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nqpfc\" (UniqueName: \"kubernetes.io/projected/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-api-access-nqpfc\") pod \"02a51eb9-3bfe-42e3-b353-afcd44834db0\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") "
Jul 28 17:48:26.951089 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.951064    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a51eb9-3bfe-42e3-b353-afcd44834db0-kserve-provision-location\") pod \"02a51eb9-3bfe-42e3-b353-afcd44834db0\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") "
Jul 28 17:48:26.951194 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.951100    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-cabundle-cert\") pod \"02a51eb9-3bfe-42e3-b353-afcd44834db0\" (UID: \"02a51eb9-3bfe-42e3-b353-afcd44834db0\") "
Jul 28 17:48:26.951315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.951290    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/02a51eb9-3bfe-42e3-b353-afcd44834db0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "02a51eb9-3bfe-42e3-b353-afcd44834db0" (UID: "02a51eb9-3bfe-42e3-b353-afcd44834db0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:26.951370 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.951292    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "02a51eb9-3bfe-42e3-b353-afcd44834db0" (UID: "02a51eb9-3bfe-42e3-b353-afcd44834db0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:26.951453 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.951423    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/02a51eb9-3bfe-42e3-b353-afcd44834db0-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:26.951453 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.951451    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:26.951634 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.951514    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "02a51eb9-3bfe-42e3-b353-afcd44834db0" (UID: "02a51eb9-3bfe-42e3-b353-afcd44834db0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:26.952962 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.952909    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/02a51eb9-3bfe-42e3-b353-afcd44834db0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "02a51eb9-3bfe-42e3-b353-afcd44834db0" (UID: "02a51eb9-3bfe-42e3-b353-afcd44834db0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:26.953052 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:26.953009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-api-access-nqpfc" (OuterVolumeSpecName: "kube-api-access-nqpfc") pod "02a51eb9-3bfe-42e3-b353-afcd44834db0" (UID: "02a51eb9-3bfe-42e3-b353-afcd44834db0"). InnerVolumeSpecName "kube-api-access-nqpfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:27.051901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.051867    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/02a51eb9-3bfe-42e3-b353-afcd44834db0-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:27.051901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.051899    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/02a51eb9-3bfe-42e3-b353-afcd44834db0-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:27.051901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.051907    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqpfc\" (UniqueName: \"kubernetes.io/projected/02a51eb9-3bfe-42e3-b353-afcd44834db0-kube-api-access-nqpfc\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:27.295999 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.295929    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv_02a51eb9-3bfe-42e3-b353-afcd44834db0/storage-initializer/1.log"
Jul 28 17:48:27.296146 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.296037    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"
Jul 28 17:48:27.296146 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.296046    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv" event={"ID":"02a51eb9-3bfe-42e3-b353-afcd44834db0","Type":"ContainerDied","Data":"1a0a144ee55d94850a7951891b5aa42d5e62b546e6190922fea672cbc04984a5"}
Jul 28 17:48:27.296146 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.296117    2563 scope.go:117] "RemoveContainer" containerID="2f59ca3cb4711c83e900d64539a866635ba74f972fdbe91b3dabf051d538e27e"
Jul 28 17:48:27.298648 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.298620    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerID="46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4" exitCode=2
Jul 28 17:48:27.298788 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.298703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" event={"ID":"6c90c8ef-65b3-4463-ac92-8628a1d506d3","Type":"ContainerDied","Data":"46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4"}
Jul 28 17:48:27.336591 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.336562    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"]
Jul 28 17:48:27.340016 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.339992    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-secondary-ede845-predictor-67bdf7dbbb-dnlbv"]
Jul 28 17:48:27.596226 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.596195    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="02a51eb9-3bfe-42e3-b353-afcd44834db0" path="/var/lib/kubelet/pods/02a51eb9-3bfe-42e3-b353-afcd44834db0/volumes"
Jul 28 17:48:27.835372 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.835332    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 28 17:48:27.839629 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:27.839599    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 28 17:48:31.052146 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.052122    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:48:31.075785 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.075760    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kserve-provision-location\") pod \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") "
Jul 28 17:48:31.075932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.075811    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90c8ef-65b3-4463-ac92-8628a1d506d3-proxy-tls\") pod \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") "
Jul 28 17:48:31.075932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.075843    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c4lb5\" (UniqueName: \"kubernetes.io/projected/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-api-access-c4lb5\") pod \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") "
Jul 28 17:48:31.075932 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.075907    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-rbac-proxy-sar-config\") pod \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") "
Jul 28 17:48:31.076106 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.076077    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-cabundle-cert\") pod \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\" (UID: \"6c90c8ef-65b3-4463-ac92-8628a1d506d3\") "
Jul 28 17:48:31.076106 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.076092    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "6c90c8ef-65b3-4463-ac92-8628a1d506d3" (UID: "6c90c8ef-65b3-4463-ac92-8628a1d506d3"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:31.076257 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.076222    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6c90c8ef-65b3-4463-ac92-8628a1d506d3" (UID: "6c90c8ef-65b3-4463-ac92-8628a1d506d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:31.076508 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.076374    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:31.076508 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.076395    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:31.076633 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.076521    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "6c90c8ef-65b3-4463-ac92-8628a1d506d3" (UID: "6c90c8ef-65b3-4463-ac92-8628a1d506d3"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:31.077956 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.077929    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6c90c8ef-65b3-4463-ac92-8628a1d506d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6c90c8ef-65b3-4463-ac92-8628a1d506d3" (UID: "6c90c8ef-65b3-4463-ac92-8628a1d506d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:31.078063 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.078013    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-api-access-c4lb5" (OuterVolumeSpecName: "kube-api-access-c4lb5") pod "6c90c8ef-65b3-4463-ac92-8628a1d506d3" (UID: "6c90c8ef-65b3-4463-ac92-8628a1d506d3"). InnerVolumeSpecName "kube-api-access-c4lb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:31.177483 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.177430    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/6c90c8ef-65b3-4463-ac92-8628a1d506d3-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:31.177483 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.177448    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6c90c8ef-65b3-4463-ac92-8628a1d506d3-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:31.177483 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.177458    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4lb5\" (UniqueName: \"kubernetes.io/projected/6c90c8ef-65b3-4463-ac92-8628a1d506d3-kube-api-access-c4lb5\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:31.316609 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.316583    2563 generic.go:356] "Generic (PLEG): container finished" podID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerID="42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5" exitCode=0
Jul 28 17:48:31.316737 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.316617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" event={"ID":"6c90c8ef-65b3-4463-ac92-8628a1d506d3","Type":"ContainerDied","Data":"42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5"}
Jul 28 17:48:31.316737 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.316641    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"
Jul 28 17:48:31.316737 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.316708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn" event={"ID":"6c90c8ef-65b3-4463-ac92-8628a1d506d3","Type":"ContainerDied","Data":"d345540926577763f59cc219a6ee0c2dc9b0d3a31eae97760961a7a1d7b7d42e"}
Jul 28 17:48:31.316901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.316749    2563 scope.go:117] "RemoveContainer" containerID="46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4"
Jul 28 17:48:31.325368 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.325351    2563 scope.go:117] "RemoveContainer" containerID="42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5"
Jul 28 17:48:31.332382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.332366    2563 scope.go:117] "RemoveContainer" containerID="fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95"
Jul 28 17:48:31.339598 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.339576    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"]
Jul 28 17:48:31.340490 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.340474    2563 scope.go:117] "RemoveContainer" containerID="46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4"
Jul 28 17:48:31.340781 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:31.340761    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4\": container with ID starting with 46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4 not found: ID does not exist" containerID="46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4"
Jul 28 17:48:31.340852 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.340788    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4"} err="failed to get container status \"46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4\": rpc error: code = NotFound desc = could not find container \"46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4\": container with ID starting with 46dea5ec19f4b14c39768f038ed7b55aa54f9fcace6ab081e59491634206dce4 not found: ID does not exist"
Jul 28 17:48:31.340852 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.340806    2563 scope.go:117] "RemoveContainer" containerID="42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5"
Jul 28 17:48:31.341071 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:31.341049    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5\": container with ID starting with 42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5 not found: ID does not exist" containerID="42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5"
Jul 28 17:48:31.341115 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.341081    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5"} err="failed to get container status \"42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5\": rpc error: code = NotFound desc = could not find container \"42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5\": container with ID starting with 42caa2677982c0ca94cdf810ba78e81a9b86518562c0f2f7114e247cd04fb4b5 not found: ID does not exist"
Jul 28 17:48:31.341115 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.341105    2563 scope.go:117] "RemoveContainer" containerID="fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95"
Jul 28 17:48:31.341345 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:31.341325    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95\": container with ID starting with fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95 not found: ID does not exist" containerID="fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95"
Jul 28 17:48:31.341397 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.341349    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95"} err="failed to get container status \"fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95\": rpc error: code = NotFound desc = could not find container \"fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95\": container with ID starting with fc43f5120d7f1e13fa1d0cd60fb8b1fd1369a171c3ab370a6a6b4ccc8d2abd95 not found: ID does not exist"
Jul 28 17:48:31.341660 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.341644    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-primary-ede845-predictor-577854c8df-5cwfn"]
Jul 28 17:48:31.595904 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.595881    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" path="/var/lib/kubelet/pods/6c90c8ef-65b3-4463-ac92-8628a1d506d3/volumes"
Jul 28 17:48:31.924870 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.924803    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"]
Jul 28 17:48:31.925095 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925083    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a51eb9-3bfe-42e3-b353-afcd44834db0" containerName="storage-initializer"
Jul 28 17:48:31.925145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925098    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a51eb9-3bfe-42e3-b353-afcd44834db0" containerName="storage-initializer"
Jul 28 17:48:31.925145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925109    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kube-rbac-proxy"
Jul 28 17:48:31.925145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925115    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kube-rbac-proxy"
Jul 28 17:48:31.925145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925133    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="02a51eb9-3bfe-42e3-b353-afcd44834db0" containerName="storage-initializer"
Jul 28 17:48:31.925145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925139    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="02a51eb9-3bfe-42e3-b353-afcd44834db0" containerName="storage-initializer"
Jul 28 17:48:31.925145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925146    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container"
Jul 28 17:48:31.925349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925152    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container"
Jul 28 17:48:31.925349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925159    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="storage-initializer"
Jul 28 17:48:31.925349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925165    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="storage-initializer"
Jul 28 17:48:31.925349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925211    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a51eb9-3bfe-42e3-b353-afcd44834db0" containerName="storage-initializer"
Jul 28 17:48:31.925349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925219    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kube-rbac-proxy"
Jul 28 17:48:31.925349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925228    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6c90c8ef-65b3-4463-ac92-8628a1d506d3" containerName="kserve-container"
Jul 28 17:48:31.925349 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.925325    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="02a51eb9-3bfe-42e3-b353-afcd44834db0" containerName="storage-initializer"
Jul 28 17:48:31.929497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.929483    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:31.932512 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.932489    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:48:31.932648 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.932468    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:48:31.932648 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.932553    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:48:31.932793 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.932783    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-secret-30a290\""
Jul 28 17:48:31.932859 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.932845    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-30a290-kube-rbac-proxy-sar-config\""
Jul 28 17:48:31.932911 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.932876    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"fail-s3-sa-30a290-dockercfg-l54lw\""
Jul 28 17:48:31.932966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.932954    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-init-fail-30a290-predictor-serving-cert\""
Jul 28 17:48:31.938042 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.938023    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"]
Jul 28 17:48:31.983708 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.983667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7hc29\" (UniqueName: \"kubernetes.io/projected/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-api-access-7hc29\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:31.983817 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.983721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:31.983817 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.983788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-cabundle-cert\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:31.983913 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.983833    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cb125eb-1812-4b91-890b-ddf4264b97ce-kserve-provision-location\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:31.983913 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:31.983871    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.084578 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.084554    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7hc29\" (UniqueName: \"kubernetes.io/projected/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-api-access-7hc29\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.084966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.084602    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.084966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.084654    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-cabundle-cert\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.084966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.084722    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cb125eb-1812-4b91-890b-ddf4264b97ce-kserve-provision-location\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.084966 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:32.084764    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-serving-cert: secret "isvc-init-fail-30a290-predictor-serving-cert" not found
Jul 28 17:48:32.084966 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.084773    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.084966 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:32.084819    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls podName:8cb125eb-1812-4b91-890b-ddf4264b97ce nodeName:}" failed. No retries permitted until 2026-07-28 17:48:32.58480081 +0000 UTC m=+1889.518200210 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls") pod "isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj" (UID: "8cb125eb-1812-4b91-890b-ddf4264b97ce") : secret "isvc-init-fail-30a290-predictor-serving-cert" not found
Jul 28 17:48:32.085312 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.085094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cb125eb-1812-4b91-890b-ddf4264b97ce-kserve-provision-location\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.085312 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.085284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-cabundle-cert\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.085481 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.085462    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-rbac-proxy-sar-config\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.093192 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.093164    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7hc29\" (UniqueName: \"kubernetes.io/projected/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-api-access-7hc29\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.587775 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.587731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.590047 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.590015    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls\") pod \"isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") " pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.839638 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.839564    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:32.966483 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:32.966459    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"]
Jul 28 17:48:32.968393 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:48:32.968364    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb125eb_1812_4b91_890b_ddf4264b97ce.slice/crio-ec9f2901fb95f2163654e1a933d9550aaba210d17dfdf6c25bd3e3c8926c1724 WatchSource:0}: Error finding container ec9f2901fb95f2163654e1a933d9550aaba210d17dfdf6c25bd3e3c8926c1724: Status 404 returned error can't find the container with id ec9f2901fb95f2163654e1a933d9550aaba210d17dfdf6c25bd3e3c8926c1724
Jul 28 17:48:33.332771 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:33.332736    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj" event={"ID":"8cb125eb-1812-4b91-890b-ddf4264b97ce","Type":"ContainerStarted","Data":"f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a"}
Jul 28 17:48:33.333244 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:33.332807    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj" event={"ID":"8cb125eb-1812-4b91-890b-ddf4264b97ce","Type":"ContainerStarted","Data":"ec9f2901fb95f2163654e1a933d9550aaba210d17dfdf6c25bd3e3c8926c1724"}
Jul 28 17:48:38.354992 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:38.354962    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj_8cb125eb-1812-4b91-890b-ddf4264b97ce/storage-initializer/0.log"
Jul 28 17:48:38.355371 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:38.354999    2563 generic.go:356] "Generic (PLEG): container finished" podID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerID="f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a" exitCode=1
Jul 28 17:48:38.355371 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:38.355058    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj" event={"ID":"8cb125eb-1812-4b91-890b-ddf4264b97ce","Type":"ContainerDied","Data":"f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a"}
Jul 28 17:48:39.360667 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:39.360636    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj_8cb125eb-1812-4b91-890b-ddf4264b97ce/storage-initializer/0.log"
Jul 28 17:48:39.361059 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:39.360711    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj" event={"ID":"8cb125eb-1812-4b91-890b-ddf4264b97ce","Type":"ContainerStarted","Data":"b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7"}
Jul 28 17:48:41.955694 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:41.955648    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"]
Jul 28 17:48:41.956152 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:41.955868    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj" podUID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerName="storage-initializer" containerID="cri-o://b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7" gracePeriod=30
Jul 28 17:48:42.696217 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.696195    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj_8cb125eb-1812-4b91-890b-ddf4264b97ce/storage-initializer/1.log"
Jul 28 17:48:42.696581 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.696565    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj_8cb125eb-1812-4b91-890b-ddf4264b97ce/storage-initializer/0.log"
Jul 28 17:48:42.698836 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.698819    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:42.765396 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765336    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-cabundle-cert\") pod \"8cb125eb-1812-4b91-890b-ddf4264b97ce\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") "
Jul 28 17:48:42.765396 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765369    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-rbac-proxy-sar-config\") pod \"8cb125eb-1812-4b91-890b-ddf4264b97ce\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") "
Jul 28 17:48:42.765576 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765411    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cb125eb-1812-4b91-890b-ddf4264b97ce-kserve-provision-location\") pod \"8cb125eb-1812-4b91-890b-ddf4264b97ce\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") "
Jul 28 17:48:42.765644 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7hc29\" (UniqueName: \"kubernetes.io/projected/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-api-access-7hc29\") pod \"8cb125eb-1812-4b91-890b-ddf4264b97ce\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") "
Jul 28 17:48:42.765644 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765626    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb125eb-1812-4b91-890b-ddf4264b97ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8cb125eb-1812-4b91-890b-ddf4264b97ce" (UID: "8cb125eb-1812-4b91-890b-ddf4264b97ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:48:42.765784 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765710    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls\") pod \"8cb125eb-1812-4b91-890b-ddf4264b97ce\" (UID: \"8cb125eb-1812-4b91-890b-ddf4264b97ce\") "
Jul 28 17:48:42.765784 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765736    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8cb125eb-1812-4b91-890b-ddf4264b97ce" (UID: "8cb125eb-1812-4b91-890b-ddf4264b97ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:42.765784 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765756    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8cb125eb-1812-4b91-890b-ddf4264b97ce" (UID: "8cb125eb-1812-4b91-890b-ddf4264b97ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:48:42.765908 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765895    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:42.765944 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765912    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:42.765944 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.765926    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8cb125eb-1812-4b91-890b-ddf4264b97ce-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:42.767589 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.767565    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-api-access-7hc29" (OuterVolumeSpecName: "kube-api-access-7hc29") pod "8cb125eb-1812-4b91-890b-ddf4264b97ce" (UID: "8cb125eb-1812-4b91-890b-ddf4264b97ce"). InnerVolumeSpecName "kube-api-access-7hc29". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:48:42.767644 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.767626    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8cb125eb-1812-4b91-890b-ddf4264b97ce" (UID: "8cb125eb-1812-4b91-890b-ddf4264b97ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:48:42.866235 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.866211    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8cb125eb-1812-4b91-890b-ddf4264b97ce-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:42.866235 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:42.866233    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7hc29\" (UniqueName: \"kubernetes.io/projected/8cb125eb-1812-4b91-890b-ddf4264b97ce-kube-api-access-7hc29\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:48:43.376650 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.376626    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj_8cb125eb-1812-4b91-890b-ddf4264b97ce/storage-initializer/1.log"
Jul 28 17:48:43.377060 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.377046    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj_8cb125eb-1812-4b91-890b-ddf4264b97ce/storage-initializer/0.log"
Jul 28 17:48:43.377102 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.377078    2563 generic.go:356] "Generic (PLEG): container finished" podID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerID="b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7" exitCode=1
Jul 28 17:48:43.377139 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.377112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj" event={"ID":"8cb125eb-1812-4b91-890b-ddf4264b97ce","Type":"ContainerDied","Data":"b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7"}
Jul 28 17:48:43.377139 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.377132    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"
Jul 28 17:48:43.377205 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.377166    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj" event={"ID":"8cb125eb-1812-4b91-890b-ddf4264b97ce","Type":"ContainerDied","Data":"ec9f2901fb95f2163654e1a933d9550aaba210d17dfdf6c25bd3e3c8926c1724"}
Jul 28 17:48:43.377205 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.377199    2563 scope.go:117] "RemoveContainer" containerID="b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7"
Jul 28 17:48:43.386416 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.386398    2563 scope.go:117] "RemoveContainer" containerID="f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a"
Jul 28 17:48:43.393472 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.393456    2563 scope.go:117] "RemoveContainer" containerID="b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7"
Jul 28 17:48:43.393732 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:43.393713    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7\": container with ID starting with b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7 not found: ID does not exist" containerID="b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7"
Jul 28 17:48:43.393793 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.393740    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7"} err="failed to get container status \"b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7\": rpc error: code = NotFound desc = could not find container \"b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7\": container with ID starting with b93612f7bc212ec77abd9c65eeb0fb27d4cf3dfd4878050dfc0ca6cd082023e7 not found: ID does not exist"
Jul 28 17:48:43.393793 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.393758    2563 scope.go:117] "RemoveContainer" containerID="f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a"
Jul 28 17:48:43.393985 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:48:43.393963    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a\": container with ID starting with f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a not found: ID does not exist" containerID="f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a"
Jul 28 17:48:43.394040 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.393994    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a"} err="failed to get container status \"f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a\": rpc error: code = NotFound desc = could not find container \"f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a\": container with ID starting with f4d20fa915e09a6ea94710b931c5daa63f988e8620f9798465c4ed6107945c9a not found: ID does not exist"
Jul 28 17:48:43.423680 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.423647    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"]
Jul 28 17:48:43.428004 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.427984    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-init-fail-30a290-predictor-7bbdbb8f5d-qcwnj"]
Jul 28 17:48:43.595954 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:43.595925    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb125eb-1812-4b91-890b-ddf4264b97ce" path="/var/lib/kubelet/pods/8cb125eb-1812-4b91-890b-ddf4264b97ce/volumes"
Jul 28 17:48:47.183848 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.183813    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"]
Jul 28 17:48:47.184196 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.184113    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerName="storage-initializer"
Jul 28 17:48:47.184196 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.184123    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerName="storage-initializer"
Jul 28 17:48:47.184196 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.184133    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerName="storage-initializer"
Jul 28 17:48:47.184196 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.184139    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerName="storage-initializer"
Jul 28 17:48:47.184196 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.184196    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerName="storage-initializer"
Jul 28 17:48:47.184408 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.184207    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb125eb-1812-4b91-890b-ddf4264b97ce" containerName="storage-initializer"
Jul 28 17:48:47.189008 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.188992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.192267 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.192242    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:48:47.193430 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.193411    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jwrfk\""
Jul 28 17:48:47.193530 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.193411    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:48:47.193530 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.193409    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:48:47.193530 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.193411    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:48:47.193691 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.193444    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-be0d8-kube-rbac-proxy-sar-config\""
Jul 28 17:48:47.193691 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.193447    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-be0d8-predictor-serving-cert\""
Jul 28 17:48:47.202458 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.202436    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"]
Jul 28 17:48:47.297830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.297801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d525e234-304e-492a-87e4-c39e988952ac-kserve-provision-location\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.297948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.297847    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d525e234-304e-492a-87e4-c39e988952ac-proxy-tls\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.297948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.297894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mg5xw\" (UniqueName: \"kubernetes.io/projected/d525e234-304e-492a-87e4-c39e988952ac-kube-api-access-mg5xw\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.297948 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.297933    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-cabundle-cert\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.298068 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.297977    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.398726 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.398692    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d525e234-304e-492a-87e4-c39e988952ac-proxy-tls\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.398901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.398731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mg5xw\" (UniqueName: \"kubernetes.io/projected/d525e234-304e-492a-87e4-c39e988952ac-kube-api-access-mg5xw\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.398901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.398762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-cabundle-cert\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.398901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.398811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.398901 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.398882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d525e234-304e-492a-87e4-c39e988952ac-kserve-provision-location\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.399270 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.399250    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d525e234-304e-492a-87e4-c39e988952ac-kserve-provision-location\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.399559 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.399540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.399702 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.399666    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-cabundle-cert\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.401362 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.401339    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d525e234-304e-492a-87e4-c39e988952ac-proxy-tls\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.407632 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.407607    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mg5xw\" (UniqueName: \"kubernetes.io/projected/d525e234-304e-492a-87e4-c39e988952ac-kube-api-access-mg5xw\") pod \"raw-sklearn-be0d8-predictor-55f497b676-7vd95\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") " pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.498715 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.498633    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:47.625291 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:47.625233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"]
Jul 28 17:48:47.627696 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:48:47.627658    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd525e234_304e_492a_87e4_c39e988952ac.slice/crio-fcb80323fb8a320601d976de26b2326b2cd87b384c129ee55155db5d6a768946 WatchSource:0}: Error finding container fcb80323fb8a320601d976de26b2326b2cd87b384c129ee55155db5d6a768946: Status 404 returned error can't find the container with id fcb80323fb8a320601d976de26b2326b2cd87b384c129ee55155db5d6a768946
Jul 28 17:48:48.398919 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:48.398887    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" event={"ID":"d525e234-304e-492a-87e4-c39e988952ac","Type":"ContainerStarted","Data":"d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8"}
Jul 28 17:48:48.399291 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:48.398942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" event={"ID":"d525e234-304e-492a-87e4-c39e988952ac","Type":"ContainerStarted","Data":"fcb80323fb8a320601d976de26b2326b2cd87b384c129ee55155db5d6a768946"}
Jul 28 17:48:52.415678 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:52.415645    2563 generic.go:356] "Generic (PLEG): container finished" podID="d525e234-304e-492a-87e4-c39e988952ac" containerID="d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8" exitCode=0
Jul 28 17:48:52.416130 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:52.415724    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" event={"ID":"d525e234-304e-492a-87e4-c39e988952ac","Type":"ContainerDied","Data":"d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8"}
Jul 28 17:48:53.420973 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:53.420938    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" event={"ID":"d525e234-304e-492a-87e4-c39e988952ac","Type":"ContainerStarted","Data":"786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8"}
Jul 28 17:48:53.421376 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:53.421025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" event={"ID":"d525e234-304e-492a-87e4-c39e988952ac","Type":"ContainerStarted","Data":"ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa"}
Jul 28 17:48:53.421376 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:53.421365    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:53.421518 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:53.421498    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:48:53.422692 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:53.422656    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:48:53.441647 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:53.441599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podStartSLOduration=2.694118686 podStartE2EDuration="6.44158583s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.747467144s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:48:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:48:53.439336374 +0000 UTC m=+1910.372735792" watchObservedRunningTime="2026-07-28 17:48:53.44158583 +0000 UTC m=+1910.374985247"
Jul 28 17:48:54.424585 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:54.424532    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:48:55.428224 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:48:55.428188    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:49:00.432662 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:49:00.432633    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:49:00.433252 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:49:00.433228    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:49:10.433934 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:49:10.433894    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:49:20.433384 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:49:20.433346    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:49:30.433917 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:49:30.433877    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:49:40.433856 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:49:40.433816    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:49:50.433415 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:49:50.433378    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:50:00.433802 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:00.433721    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:50:10.433789 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:10.433758    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:50:17.292081 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:17.292050    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"]
Jul 28 17:50:17.292596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:17.292298    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" containerID="cri-o://ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa" gracePeriod=30
Jul 28 17:50:17.292596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:17.292305    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kube-rbac-proxy" containerID="cri-o://786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8" gracePeriod=30
Jul 28 17:50:17.746338 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:17.746303    2563 generic.go:356] "Generic (PLEG): container finished" podID="d525e234-304e-492a-87e4-c39e988952ac" containerID="786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8" exitCode=2
Jul 28 17:50:17.746493 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:17.746343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" event={"ID":"d525e234-304e-492a-87e4-c39e988952ac","Type":"ContainerDied","Data":"786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8"}
Jul 28 17:50:20.428931 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:20.428890    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 28 17:50:20.433370 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:20.433342    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 28 17:50:21.433696 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.433659    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:50:21.513285 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.513221    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mg5xw\" (UniqueName: \"kubernetes.io/projected/d525e234-304e-492a-87e4-c39e988952ac-kube-api-access-mg5xw\") pod \"d525e234-304e-492a-87e4-c39e988952ac\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") "
Jul 28 17:50:21.513285 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.513262    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-cabundle-cert\") pod \"d525e234-304e-492a-87e4-c39e988952ac\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") "
Jul 28 17:50:21.513285 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.513285    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d525e234-304e-492a-87e4-c39e988952ac-proxy-tls\") pod \"d525e234-304e-492a-87e4-c39e988952ac\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") "
Jul 28 17:50:21.513502 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.513350    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-kube-rbac-proxy-sar-config\") pod \"d525e234-304e-492a-87e4-c39e988952ac\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") "
Jul 28 17:50:21.513502 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.513395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d525e234-304e-492a-87e4-c39e988952ac-kserve-provision-location\") pod \"d525e234-304e-492a-87e4-c39e988952ac\" (UID: \"d525e234-304e-492a-87e4-c39e988952ac\") "
Jul 28 17:50:21.513623 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.513557    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d525e234-304e-492a-87e4-c39e988952ac" (UID: "d525e234-304e-492a-87e4-c39e988952ac"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:50:21.513808 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.513786    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d525e234-304e-492a-87e4-c39e988952ac-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d525e234-304e-492a-87e4-c39e988952ac" (UID: "d525e234-304e-492a-87e4-c39e988952ac"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:50:21.513808 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.513788    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d525e234-304e-492a-87e4-c39e988952ac" (UID: "d525e234-304e-492a-87e4-c39e988952ac"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:50:21.515228 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.515206    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d525e234-304e-492a-87e4-c39e988952ac-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d525e234-304e-492a-87e4-c39e988952ac" (UID: "d525e234-304e-492a-87e4-c39e988952ac"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:50:21.515327 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.515270    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d525e234-304e-492a-87e4-c39e988952ac-kube-api-access-mg5xw" (OuterVolumeSpecName: "kube-api-access-mg5xw") pod "d525e234-304e-492a-87e4-c39e988952ac" (UID: "d525e234-304e-492a-87e4-c39e988952ac"). InnerVolumeSpecName "kube-api-access-mg5xw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:50:21.614325 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.614306    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mg5xw\" (UniqueName: \"kubernetes.io/projected/d525e234-304e-492a-87e4-c39e988952ac-kube-api-access-mg5xw\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:50:21.614422 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.614327    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:50:21.614422 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.614337    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d525e234-304e-492a-87e4-c39e988952ac-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:50:21.614422 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.614345    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d525e234-304e-492a-87e4-c39e988952ac-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:50:21.614422 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.614354    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d525e234-304e-492a-87e4-c39e988952ac-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:50:21.764295 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.764225    2563 generic.go:356] "Generic (PLEG): container finished" podID="d525e234-304e-492a-87e4-c39e988952ac" containerID="ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa" exitCode=0
Jul 28 17:50:21.764295 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.764289    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"
Jul 28 17:50:21.764445 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.764289    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" event={"ID":"d525e234-304e-492a-87e4-c39e988952ac","Type":"ContainerDied","Data":"ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa"}
Jul 28 17:50:21.764445 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.764363    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95" event={"ID":"d525e234-304e-492a-87e4-c39e988952ac","Type":"ContainerDied","Data":"fcb80323fb8a320601d976de26b2326b2cd87b384c129ee55155db5d6a768946"}
Jul 28 17:50:21.764445 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.764391    2563 scope.go:117] "RemoveContainer" containerID="786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8"
Jul 28 17:50:21.773422 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.773401    2563 scope.go:117] "RemoveContainer" containerID="ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa"
Jul 28 17:50:21.780451 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.780430    2563 scope.go:117] "RemoveContainer" containerID="d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8"
Jul 28 17:50:21.783652 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.783633    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"]
Jul 28 17:50:21.787277 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.787259    2563 scope.go:117] "RemoveContainer" containerID="786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8"
Jul 28 17:50:21.787520 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:50:21.787503    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8\": container with ID starting with 786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8 not found: ID does not exist" containerID="786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8"
Jul 28 17:50:21.787567 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.787526    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8"} err="failed to get container status \"786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8\": rpc error: code = NotFound desc = could not find container \"786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8\": container with ID starting with 786ef51c4a016a36dc1819d65d694fd57baa1353752c3991df8ed4779bed3ca8 not found: ID does not exist"
Jul 28 17:50:21.787567 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.787541    2563 scope.go:117] "RemoveContainer" containerID="ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa"
Jul 28 17:50:21.787757 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:50:21.787737    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa\": container with ID starting with ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa not found: ID does not exist" containerID="ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa"
Jul 28 17:50:21.787811 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.787767    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa"} err="failed to get container status \"ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa\": rpc error: code = NotFound desc = could not find container \"ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa\": container with ID starting with ac52fe903fbf6ef53d79145c2d28a6138f8461b407e415a79ac548953a7daeaa not found: ID does not exist"
Jul 28 17:50:21.787811 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.787789    2563 scope.go:117] "RemoveContainer" containerID="d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8"
Jul 28 17:50:21.788021 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:50:21.788003    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8\": container with ID starting with d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8 not found: ID does not exist" containerID="d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8"
Jul 28 17:50:21.788082 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.788026    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8"} err="failed to get container status \"d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8\": rpc error: code = NotFound desc = could not find container \"d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8\": container with ID starting with d4622e650796593db0e887aa4c3bb5cd63230d83d735efe749f247c5cd48d8c8 not found: ID does not exist"
Jul 28 17:50:21.790293 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:21.790273    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-be0d8-predictor-55f497b676-7vd95"]
Jul 28 17:50:22.498254 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498224    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"]
Jul 28 17:50:22.498755 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498740    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kube-rbac-proxy"
Jul 28 17:50:22.498830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498759    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kube-rbac-proxy"
Jul 28 17:50:22.498830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498775    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container"
Jul 28 17:50:22.498830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498784    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container"
Jul 28 17:50:22.498830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498798    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="storage-initializer"
Jul 28 17:50:22.498830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498808    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="storage-initializer"
Jul 28 17:50:22.499095 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498891    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kube-rbac-proxy"
Jul 28 17:50:22.499095 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.498902    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d525e234-304e-492a-87e4-c39e988952ac" containerName="kserve-container"
Jul 28 17:50:22.503400 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.503380    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.507156 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.507136    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-8e468-kube-rbac-proxy-sar-config\""
Jul 28 17:50:22.507156 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.507152    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 17:50:22.507302 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.507136    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 17:50:22.508464 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.508445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"raw-sklearn-runtime-8e468-predictor-serving-cert\""
Jul 28 17:50:22.508606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.508445    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 28 17:50:22.508606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.508448    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jwrfk\""
Jul 28 17:50:22.508606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.508469    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 28 17:50:22.514070 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.514050    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"]
Jul 28 17:50:22.620743 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.620718    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.620886 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.620747    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spgmj\" (UniqueName: \"kubernetes.io/projected/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-api-access-spgmj\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.620886 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.620801    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kserve-provision-location\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.620886 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.620826    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-cabundle-cert\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.620886 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.620859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-proxy-tls\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.721166 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.721133    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kserve-provision-location\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.721166 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.721169    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-cabundle-cert\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.721295 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.721195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-proxy-tls\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.721295 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.721226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.721295 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.721243    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spgmj\" (UniqueName: \"kubernetes.io/projected/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-api-access-spgmj\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.721601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.721561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kserve-provision-location\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.721906 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.721884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-rbac-proxy-sar-config\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.722010 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.721907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-cabundle-cert\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.723647 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.723625    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-proxy-tls\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.730516 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.730496    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-spgmj\" (UniqueName: \"kubernetes.io/projected/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-api-access-spgmj\") pod \"raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") " pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.813878 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.813820    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:22.944936 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:22.944906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"]
Jul 28 17:50:22.947993 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:50:22.947963    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3d9f68f_ed58_4773_961c_5bcf5d193a7a.slice/crio-e7681ced1cb18b987fff7a9ada0a539d9f576b8a1ca9e455854aba4c58b6b10e WatchSource:0}: Error finding container e7681ced1cb18b987fff7a9ada0a539d9f576b8a1ca9e455854aba4c58b6b10e: Status 404 returned error can't find the container with id e7681ced1cb18b987fff7a9ada0a539d9f576b8a1ca9e455854aba4c58b6b10e
Jul 28 17:50:23.596879 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:23.596845    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d525e234-304e-492a-87e4-c39e988952ac" path="/var/lib/kubelet/pods/d525e234-304e-492a-87e4-c39e988952ac/volumes"
Jul 28 17:50:23.775120 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:23.775079    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" event={"ID":"d3d9f68f-ed58-4773-961c-5bcf5d193a7a","Type":"ContainerStarted","Data":"61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386"}
Jul 28 17:50:23.775312 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:23.775151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" event={"ID":"d3d9f68f-ed58-4773-961c-5bcf5d193a7a","Type":"ContainerStarted","Data":"e7681ced1cb18b987fff7a9ada0a539d9f576b8a1ca9e455854aba4c58b6b10e"}
Jul 28 17:50:26.789494 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:26.789463    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerID="61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386" exitCode=0
Jul 28 17:50:26.789806 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:26.789537    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" event={"ID":"d3d9f68f-ed58-4773-961c-5bcf5d193a7a","Type":"ContainerDied","Data":"61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386"}
Jul 28 17:50:27.795996 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:27.795957    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" event={"ID":"d3d9f68f-ed58-4773-961c-5bcf5d193a7a","Type":"ContainerStarted","Data":"ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0"}
Jul 28 17:50:27.796428 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:27.796023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" event={"ID":"d3d9f68f-ed58-4773-961c-5bcf5d193a7a","Type":"ContainerStarted","Data":"ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e"}
Jul 28 17:50:27.796428 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:27.796317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:27.796512 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:27.796480    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:27.797629 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:27.797601    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:50:27.817867 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:27.817822    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podStartSLOduration=2.1419169 podStartE2EDuration="5.817808454s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.675891554s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:50:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:50:27.816266871 +0000 UTC m=+2004.749666314" watchObservedRunningTime="2026-07-28 17:50:27.817808454 +0000 UTC m=+2004.751207872"
Jul 28 17:50:28.799688 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:28.799631    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:50:33.803940 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:33.803909    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:50:33.804396 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:33.804368    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:50:43.804390 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:43.804344    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:50:53.804873 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:50:53.804836    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:51:03.804583 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:03.804544    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:51:13.805102 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:13.805067    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:51:23.804601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:23.804564    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:51:33.804468 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:33.804387    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:51:43.805839 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:43.805810    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:51:52.560367 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:52.560334    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"]
Jul 28 17:51:52.560897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:52.560554    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" containerID="cri-o://ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e" gracePeriod=30
Jul 28 17:51:52.560897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:52.560587    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kube-rbac-proxy" containerID="cri-o://ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0" gracePeriod=30
Jul 28 17:51:53.137485 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:53.137452    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerID="ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0" exitCode=2
Jul 28 17:51:53.137651 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:53.137521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" event={"ID":"d3d9f68f-ed58-4773-961c-5bcf5d193a7a","Type":"ContainerDied","Data":"ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0"}
Jul 28 17:51:53.800685 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:53.800633    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 28 17:51:53.804325 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:53.804297    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 28 17:51:56.608302 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.608275    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:51:56.642315 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.642291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-proxy-tls\") pod \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") "
Jul 28 17:51:56.642461 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.642323    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-cabundle-cert\") pod \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") "
Jul 28 17:51:56.642461 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.642351    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-rbac-proxy-sar-config\") pod \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") "
Jul 28 17:51:56.642461 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.642381    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kserve-provision-location\") pod \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") "
Jul 28 17:51:56.642461 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.642399    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-spgmj\" (UniqueName: \"kubernetes.io/projected/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-api-access-spgmj\") pod \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\" (UID: \"d3d9f68f-ed58-4773-961c-5bcf5d193a7a\") "
Jul 28 17:51:56.642740 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.642718    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d3d9f68f-ed58-4773-961c-5bcf5d193a7a" (UID: "d3d9f68f-ed58-4773-961c-5bcf5d193a7a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:56.642807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.642730    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d3d9f68f-ed58-4773-961c-5bcf5d193a7a" (UID: "d3d9f68f-ed58-4773-961c-5bcf5d193a7a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:51:56.642807 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.642748    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d3d9f68f-ed58-4773-961c-5bcf5d193a7a" (UID: "d3d9f68f-ed58-4773-961c-5bcf5d193a7a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 17:51:56.644501 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.644469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-api-access-spgmj" (OuterVolumeSpecName: "kube-api-access-spgmj") pod "d3d9f68f-ed58-4773-961c-5bcf5d193a7a" (UID: "d3d9f68f-ed58-4773-961c-5bcf5d193a7a"). InnerVolumeSpecName "kube-api-access-spgmj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:51:56.644588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.644540    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d3d9f68f-ed58-4773-961c-5bcf5d193a7a" (UID: "d3d9f68f-ed58-4773-961c-5bcf5d193a7a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 17:51:56.744045 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.743946    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:51:56.744045 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.743990    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kserve-provision-location\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:51:56.744045 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.744001    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spgmj\" (UniqueName: \"kubernetes.io/projected/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-kube-api-access-spgmj\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:51:56.744045 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.744010    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-proxy-tls\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:51:56.744045 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:56.744022    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d3d9f68f-ed58-4773-961c-5bcf5d193a7a-cabundle-cert\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:51:57.155331 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.155300    2563 generic.go:356] "Generic (PLEG): container finished" podID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerID="ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e" exitCode=0
Jul 28 17:51:57.155497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.155347    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"
Jul 28 17:51:57.155497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.155353    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" event={"ID":"d3d9f68f-ed58-4773-961c-5bcf5d193a7a","Type":"ContainerDied","Data":"ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e"}
Jul 28 17:51:57.155497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.155406    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs" event={"ID":"d3d9f68f-ed58-4773-961c-5bcf5d193a7a","Type":"ContainerDied","Data":"e7681ced1cb18b987fff7a9ada0a539d9f576b8a1ca9e455854aba4c58b6b10e"}
Jul 28 17:51:57.155497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.155434    2563 scope.go:117] "RemoveContainer" containerID="ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0"
Jul 28 17:51:57.163455 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.163272    2563 scope.go:117] "RemoveContainer" containerID="ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e"
Jul 28 17:51:57.170511 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.170496    2563 scope.go:117] "RemoveContainer" containerID="61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386"
Jul 28 17:51:57.177794 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.177776    2563 scope.go:117] "RemoveContainer" containerID="ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0"
Jul 28 17:51:57.178086 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:51:57.178065    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0\": container with ID starting with ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0 not found: ID does not exist" containerID="ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0"
Jul 28 17:51:57.178145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.178081    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"]
Jul 28 17:51:57.178145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.178093    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0"} err="failed to get container status \"ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0\": rpc error: code = NotFound desc = could not find container \"ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0\": container with ID starting with ca6c074642897ec6362ad974cbb01661cd19acf810f0d0d4913da409d39896e0 not found: ID does not exist"
Jul 28 17:51:57.178145 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.178110    2563 scope.go:117] "RemoveContainer" containerID="ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e"
Jul 28 17:51:57.178358 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:51:57.178337    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e\": container with ID starting with ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e not found: ID does not exist" containerID="ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e"
Jul 28 17:51:57.178396 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.178356    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e"} err="failed to get container status \"ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e\": rpc error: code = NotFound desc = could not find container \"ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e\": container with ID starting with ff9d1a6a386e3b02cb5188cc6de4774b2210ce4d9e79982bfb9c9fcd60dae66e not found: ID does not exist"
Jul 28 17:51:57.178396 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.178372    2563 scope.go:117] "RemoveContainer" containerID="61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386"
Jul 28 17:51:57.178557 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:51:57.178539    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386\": container with ID starting with 61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386 not found: ID does not exist" containerID="61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386"
Jul 28 17:51:57.178596 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.178561    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386"} err="failed to get container status \"61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386\": rpc error: code = NotFound desc = could not find container \"61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386\": container with ID starting with 61e3fcc9b4d93d0714ffb41160a94e64762ea2653ae759f9d3a46b90cacc6386 not found: ID does not exist"
Jul 28 17:51:57.184735 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.184664    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/raw-sklearn-runtime-8e468-predictor-845d8647f6-9xcqs"]
Jul 28 17:51:57.596480 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:57.596450    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" path="/var/lib/kubelet/pods/d3d9f68f-ed58-4773-961c-5bcf5d193a7a/volumes"
Jul 28 17:51:59.200416 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200376    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-55c8s/must-gather-96r2q"]
Jul 28 17:51:59.200897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200704    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kube-rbac-proxy"
Jul 28 17:51:59.200897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200721    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kube-rbac-proxy"
Jul 28 17:51:59.200897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200733    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container"
Jul 28 17:51:59.200897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200741    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container"
Jul 28 17:51:59.200897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200748    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="storage-initializer"
Jul 28 17:51:59.200897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200754    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="storage-initializer"
Jul 28 17:51:59.200897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200808    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kserve-container"
Jul 28 17:51:59.200897 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.200818    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3d9f68f-ed58-4773-961c-5bcf5d193a7a" containerName="kube-rbac-proxy"
Jul 28 17:51:59.204989 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.204970    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:51:59.208272 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.208247    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-55c8s\"/\"openshift-service-ca.crt\""
Jul 28 17:51:59.208378 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.208333    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-55c8s\"/\"default-dockercfg-xz66p\""
Jul 28 17:51:59.209830 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.209816    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-55c8s\"/\"kube-root-ca.crt\""
Jul 28 17:51:59.218888 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.218866    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-55c8s/must-gather-96r2q"]
Jul 28 17:51:59.263750 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.263727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-must-gather-output\") pod \"must-gather-96r2q\" (UID: \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\") " pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:51:59.263841 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.263782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txvf6\" (UniqueName: \"kubernetes.io/projected/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-kube-api-access-txvf6\") pod \"must-gather-96r2q\" (UID: \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\") " pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:51:59.364523 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.364501    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txvf6\" (UniqueName: \"kubernetes.io/projected/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-kube-api-access-txvf6\") pod \"must-gather-96r2q\" (UID: \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\") " pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:51:59.364601 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.364541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-must-gather-output\") pod \"must-gather-96r2q\" (UID: \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\") " pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:51:59.364847 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.364833    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-must-gather-output\") pod \"must-gather-96r2q\" (UID: \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\") " pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:51:59.374929 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.374907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-txvf6\" (UniqueName: \"kubernetes.io/projected/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-kube-api-access-txvf6\") pod \"must-gather-96r2q\" (UID: \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\") " pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:51:59.531220 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.531170    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:51:59.661400 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.661369    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-55c8s/must-gather-96r2q"]
Jul 28 17:51:59.663995 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:51:59.663956    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f8b0f24_9d76_4829_94b0_6d5fe7795ad2.slice/crio-644cb7a7ebf818efb9c0465740f479721b4bc2589f9a9c3bd8f6b2d00df84bcd WatchSource:0}: Error finding container 644cb7a7ebf818efb9c0465740f479721b4bc2589f9a9c3bd8f6b2d00df84bcd: Status 404 returned error can't find the container with id 644cb7a7ebf818efb9c0465740f479721b4bc2589f9a9c3bd8f6b2d00df84bcd
Jul 28 17:51:59.666077 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:51:59.666062    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 17:52:00.167081 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:00.167048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55c8s/must-gather-96r2q" event={"ID":"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2","Type":"ContainerStarted","Data":"644cb7a7ebf818efb9c0465740f479721b4bc2589f9a9c3bd8f6b2d00df84bcd"}
Jul 28 17:52:06.197535 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:06.197498    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55c8s/must-gather-96r2q" event={"ID":"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2","Type":"ContainerStarted","Data":"c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819"}
Jul 28 17:52:06.197535 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:06.197540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55c8s/must-gather-96r2q" event={"ID":"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2","Type":"ContainerStarted","Data":"5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb"}
Jul 28 17:52:06.214728 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:06.214683    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-55c8s/must-gather-96r2q" podStartSLOduration=1.218200788 podStartE2EDuration="7.214656624s" totalImagesPullingTime="5.996455836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:51:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:06.213907116 +0000 UTC m=+2103.147306534" watchObservedRunningTime="2026-07-28 17:52:06.214656624 +0000 UTC m=+2103.148056041"
Jul 28 17:52:28.288046 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.288013    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerID="5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb" exitCode=0
Jul 28 17:52:28.288450 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.288085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-55c8s/must-gather-96r2q" event={"ID":"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2","Type":"ContainerDied","Data":"5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb"}
Jul 28 17:52:28.288450 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.288390    2563 scope.go:117] "RemoveContainer" containerID="5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb"
Jul 28 17:52:28.331760 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.331735    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-55c8s_must-gather-96r2q_3f8b0f24-9d76-4829-94b0-6d5fe7795ad2/gather/0.log"
Jul 28 17:52:28.905653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.905623    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cjjwk/must-gather-z996c"]
Jul 28 17:52:28.908124 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.908100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjjwk/must-gather-z996c"
Jul 28 17:52:28.912103 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.912074    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjjwk\"/\"kube-root-ca.crt\""
Jul 28 17:52:28.912296 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.912107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cjjwk\"/\"openshift-service-ca.crt\""
Jul 28 17:52:28.913408 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.913386    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cjjwk\"/\"default-dockercfg-dvp87\""
Jul 28 17:52:28.921067 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:28.921045    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjjwk/must-gather-z996c"]
Jul 28 17:52:29.027920 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:29.027889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/790ea860-f2e7-4be3-b34a-6fa890e5675a-must-gather-output\") pod \"must-gather-z996c\" (UID: \"790ea860-f2e7-4be3-b34a-6fa890e5675a\") " pod="openshift-must-gather-cjjwk/must-gather-z996c"
Jul 28 17:52:29.028051 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:29.027935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mgn49\" (UniqueName: \"kubernetes.io/projected/790ea860-f2e7-4be3-b34a-6fa890e5675a-kube-api-access-mgn49\") pod \"must-gather-z996c\" (UID: \"790ea860-f2e7-4be3-b34a-6fa890e5675a\") " pod="openshift-must-gather-cjjwk/must-gather-z996c"
Jul 28 17:52:29.128347 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:29.128316    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/790ea860-f2e7-4be3-b34a-6fa890e5675a-must-gather-output\") pod \"must-gather-z996c\" (UID: \"790ea860-f2e7-4be3-b34a-6fa890e5675a\") " pod="openshift-must-gather-cjjwk/must-gather-z996c"
Jul 28 17:52:29.128447 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:29.128369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mgn49\" (UniqueName: \"kubernetes.io/projected/790ea860-f2e7-4be3-b34a-6fa890e5675a-kube-api-access-mgn49\") pod \"must-gather-z996c\" (UID: \"790ea860-f2e7-4be3-b34a-6fa890e5675a\") " pod="openshift-must-gather-cjjwk/must-gather-z996c"
Jul 28 17:52:29.128725 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:29.128709    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/790ea860-f2e7-4be3-b34a-6fa890e5675a-must-gather-output\") pod \"must-gather-z996c\" (UID: \"790ea860-f2e7-4be3-b34a-6fa890e5675a\") " pod="openshift-must-gather-cjjwk/must-gather-z996c"
Jul 28 17:52:29.136926 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:29.136907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mgn49\" (UniqueName: \"kubernetes.io/projected/790ea860-f2e7-4be3-b34a-6fa890e5675a-kube-api-access-mgn49\") pod \"must-gather-z996c\" (UID: \"790ea860-f2e7-4be3-b34a-6fa890e5675a\") " pod="openshift-must-gather-cjjwk/must-gather-z996c"
Jul 28 17:52:29.216994 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:29.216943    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjjwk/must-gather-z996c"
Jul 28 17:52:29.339588 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:29.339565    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjjwk/must-gather-z996c"]
Jul 28 17:52:29.342234 ip-10-0-137-64 kubenswrapper[2563]: W0728 17:52:29.342200    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod790ea860_f2e7_4be3_b34a_6fa890e5675a.slice/crio-539e4e0a06ae00ed5f4089b124e0cc1bc0999016de70dcb495f94367e90241aa WatchSource:0}: Error finding container 539e4e0a06ae00ed5f4089b124e0cc1bc0999016de70dcb495f94367e90241aa: Status 404 returned error can't find the container with id 539e4e0a06ae00ed5f4089b124e0cc1bc0999016de70dcb495f94367e90241aa
Jul 28 17:52:30.298326 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:30.298249    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjjwk/must-gather-z996c" event={"ID":"790ea860-f2e7-4be3-b34a-6fa890e5675a","Type":"ContainerStarted","Data":"8513b335cc5491873eea4a3903693ab3b4009d66eeb711fdc4e49bcc48b3e584"}
Jul 28 17:52:30.298326 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:30.298299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjjwk/must-gather-z996c" event={"ID":"790ea860-f2e7-4be3-b34a-6fa890e5675a","Type":"ContainerStarted","Data":"4932646b9898d94c104f6ab78d6d90a6464de74cbf0b07dc0c739941941bf848"}
Jul 28 17:52:30.298326 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:30.298314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjjwk/must-gather-z996c" event={"ID":"790ea860-f2e7-4be3-b34a-6fa890e5675a","Type":"ContainerStarted","Data":"539e4e0a06ae00ed5f4089b124e0cc1bc0999016de70dcb495f94367e90241aa"}
Jul 28 17:52:30.317183 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:30.317133    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cjjwk/must-gather-z996c" podStartSLOduration=1.6292377390000001 podStartE2EDuration="2.317117773s" totalImagesPullingTime="687.880034ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 17:52:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:30.316921488 +0000 UTC m=+2127.250320906" watchObservedRunningTime="2026-07-28 17:52:30.317117773 +0000 UTC m=+2127.250517190"
Jul 28 17:52:31.476598 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:31.476568    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gjl4f_42b3b5e9-0f43-4c50-be37-a7c61ff99746/global-pull-secret-syncer/0.log"
Jul 28 17:52:31.599580 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:31.599552    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cn5vl_767e90f1-3aaa-4ec1-adb0-e732020844d5/konnectivity-agent/0.log"
Jul 28 17:52:31.678967 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:31.678935    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-137-64.ec2.internal_2557c4e520af9150ace36fc6434afd7f/haproxy/0.log"
Jul 28 17:52:33.772974 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:33.772934    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-55c8s/must-gather-96r2q"]
Jul 28 17:52:33.773824 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:33.773791    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-55c8s/must-gather-96r2q" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerName="copy" containerID="cri-o://c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819" gracePeriod=2
Jul 28 17:52:33.775358 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:33.775187    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-55c8s/must-gather-96r2q"]
Jul 28 17:52:33.778093 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:52:33.778055    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-96r2q\" is forbidden: User \"system:node:ip-10-0-137-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-55c8s\": no relationship found between node 'ip-10-0-137-64.ec2.internal' and this object" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:52:34.117491 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.115718    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-55c8s_must-gather-96r2q_3f8b0f24-9d76-4829-94b0-6d5fe7795ad2/copy/0.log"
Jul 28 17:52:34.121497 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.121197    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:52:34.125689 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:52:34.124909    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-96r2q\" is forbidden: User \"system:node:ip-10-0-137-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-55c8s\": no relationship found between node 'ip-10-0-137-64.ec2.internal' and this object" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:52:34.283578 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.282889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-must-gather-output\") pod \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\" (UID: \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\") "
Jul 28 17:52:34.283578 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.282975    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-txvf6\" (UniqueName: \"kubernetes.io/projected/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-kube-api-access-txvf6\") pod \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\" (UID: \"3f8b0f24-9d76-4829-94b0-6d5fe7795ad2\") "
Jul 28 17:52:34.288388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.285125    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" (UID: "3f8b0f24-9d76-4829-94b0-6d5fe7795ad2"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 28 17:52:34.294218 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.294181    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-kube-api-access-txvf6" (OuterVolumeSpecName: "kube-api-access-txvf6") pod "3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" (UID: "3f8b0f24-9d76-4829-94b0-6d5fe7795ad2"). InnerVolumeSpecName "kube-api-access-txvf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 17:52:34.320813 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.320780    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-55c8s_must-gather-96r2q_3f8b0f24-9d76-4829-94b0-6d5fe7795ad2/copy/0.log"
Jul 28 17:52:34.325745 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.321188    2563 generic.go:356] "Generic (PLEG): container finished" podID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerID="c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819" exitCode=143
Jul 28 17:52:34.325745 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.321330    2563 scope.go:117] "RemoveContainer" containerID="c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819"
Jul 28 17:52:34.325745 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.321506    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:52:34.329153 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:52:34.329085    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-96r2q\" is forbidden: User \"system:node:ip-10-0-137-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-55c8s\": no relationship found between node 'ip-10-0-137-64.ec2.internal' and this object" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:52:34.346816 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.346515    2563 scope.go:117] "RemoveContainer" containerID="5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb"
Jul 28 17:52:34.360693 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:52:34.356869    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-96r2q\" is forbidden: User \"system:node:ip-10-0-137-64.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-55c8s\": no relationship found between node 'ip-10-0-137-64.ec2.internal' and this object" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" pod="openshift-must-gather-55c8s/must-gather-96r2q"
Jul 28 17:52:34.372394 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.371367    2563 scope.go:117] "RemoveContainer" containerID="c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819"
Jul 28 17:52:34.372394 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:52:34.371730    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819\": container with ID starting with c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819 not found: ID does not exist" containerID="c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819"
Jul 28 17:52:34.372394 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.371763    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819"} err="failed to get container status \"c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819\": rpc error: code = NotFound desc = could not find container \"c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819\": container with ID starting with c8d08a9c964237588fe56ef1f93c195c744026a16ae20966953ce504d2a4f819 not found: ID does not exist"
Jul 28 17:52:34.372394 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.371784    2563 scope.go:117] "RemoveContainer" containerID="5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb"
Jul 28 17:52:34.372394 ip-10-0-137-64 kubenswrapper[2563]: E0728 17:52:34.372104    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb\": container with ID starting with 5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb not found: ID does not exist" containerID="5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb"
Jul 28 17:52:34.372394 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.372127    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb"} err="failed to get container status \"5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb\": rpc error: code = NotFound desc = could not find container \"5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb\": container with ID starting with 5444014ad6aae6eba97ee65e8b5b2a6681ddc3dafe8e8f2a0086c8f2ede6d7fb not found: ID does not exist"
Jul 28 17:52:34.384006 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.383975    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-must-gather-output\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:52:34.384006 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:34.384007    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txvf6\" (UniqueName: \"kubernetes.io/projected/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2-kube-api-access-txvf6\") on node \"ip-10-0-137-64.ec2.internal\" DevicePath \"\""
Jul 28 17:52:35.332181 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.332109    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f77bfc2-677a-4fa6-baeb-4b13150d76c7/alertmanager/0.log"
Jul 28 17:52:35.369092 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.369061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f77bfc2-677a-4fa6-baeb-4b13150d76c7/config-reloader/0.log"
Jul 28 17:52:35.399660 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.399626    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f77bfc2-677a-4fa6-baeb-4b13150d76c7/kube-rbac-proxy-web/0.log"
Jul 28 17:52:35.433910 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.433833    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f77bfc2-677a-4fa6-baeb-4b13150d76c7/kube-rbac-proxy/0.log"
Jul 28 17:52:35.465382 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.465344    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f77bfc2-677a-4fa6-baeb-4b13150d76c7/kube-rbac-proxy-metric/0.log"
Jul 28 17:52:35.504715 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.504690    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f77bfc2-677a-4fa6-baeb-4b13150d76c7/prom-label-proxy/0.log"
Jul 28 17:52:35.537039 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.536997    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_5f77bfc2-677a-4fa6-baeb-4b13150d76c7/init-config-reloader/0.log"
Jul 28 17:52:35.601411 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.601378    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" path="/var/lib/kubelet/pods/3f8b0f24-9d76-4829-94b0-6d5fe7795ad2/volumes"
Jul 28 17:52:35.784908 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.784800    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-77fcdc95f7-dg5d4_1b24d701-1478-47b1-b8da-839a8fe0034a/metrics-server/0.log"
Jul 28 17:52:35.858236 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.858208    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fgpvf_08dc8e97-c2e1-4238-b172-a40801fca1a8/node-exporter/0.log"
Jul 28 17:52:35.889070 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.889040    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fgpvf_08dc8e97-c2e1-4238-b172-a40801fca1a8/kube-rbac-proxy/0.log"
Jul 28 17:52:35.924510 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:35.924475    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-fgpvf_08dc8e97-c2e1-4238-b172-a40801fca1a8/init-textfile/0.log"
Jul 28 17:52:36.268004 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.267968    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4da263b-494e-4aa0-b450-4fe5c26d4430/prometheus/0.log"
Jul 28 17:52:36.294649 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.294610    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4da263b-494e-4aa0-b450-4fe5c26d4430/config-reloader/0.log"
Jul 28 17:52:36.316532 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.316506    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4da263b-494e-4aa0-b450-4fe5c26d4430/thanos-sidecar/0.log"
Jul 28 17:52:36.339207 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.339185    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4da263b-494e-4aa0-b450-4fe5c26d4430/kube-rbac-proxy-web/0.log"
Jul 28 17:52:36.363442 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.363413    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4da263b-494e-4aa0-b450-4fe5c26d4430/kube-rbac-proxy/0.log"
Jul 28 17:52:36.390320 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.390287    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4da263b-494e-4aa0-b450-4fe5c26d4430/kube-rbac-proxy-thanos/0.log"
Jul 28 17:52:36.418782 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.418755    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4da263b-494e-4aa0-b450-4fe5c26d4430/init-config-reloader/0.log"
Jul 28 17:52:36.456866 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.456828    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-2484n_b2abc20c-0068-4bbc-886f-75228cca03ac/prometheus-operator/0.log"
Jul 28 17:52:36.484480 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.484444    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-84d69f794d-2484n_b2abc20c-0068-4bbc-886f-75228cca03ac/kube-rbac-proxy/0.log"
Jul 28 17:52:36.560260 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.560172    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57c4bcdbbb-wngtq_b544baa2-b772-4e08-88d0-b0a42bdb3070/telemeter-client/0.log"
Jul 28 17:52:36.583434 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.583407    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57c4bcdbbb-wngtq_b544baa2-b772-4e08-88d0-b0a42bdb3070/reload/0.log"
Jul 28 17:52:36.608384 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:36.608353    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-57c4bcdbbb-wngtq_b544baa2-b772-4e08-88d0-b0a42bdb3070/kube-rbac-proxy/0.log"
Jul 28 17:52:39.139372 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.139337    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-7c97dc4555-sp2kw_da8735fd-e062-458c-933f-7d344a34f479/download-server/0.log"
Jul 28 17:52:39.729410 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.729378    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"]
Jul 28 17:52:39.729734 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.729717    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerName="gather"
Jul 28 17:52:39.729879 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.729863    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerName="gather"
Jul 28 17:52:39.729955 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.729888    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerName="copy"
Jul 28 17:52:39.729955 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.729897    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerName="copy"
Jul 28 17:52:39.730057 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.729998    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerName="gather"
Jul 28 17:52:39.730057 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.730011    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f8b0f24-9d76-4829-94b0-6d5fe7795ad2" containerName="copy"
Jul 28 17:52:39.735870 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.735844    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.741715 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.741684    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"]
Jul 28 17:52:39.846701 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.846654    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-sys\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.846887 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.846734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-proc\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.846887 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.846752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-lib-modules\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.846887 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.846803    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-podres\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.846887 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.846876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8cm5p\" (UniqueName: \"kubernetes.io/projected/c17947c1-bb33-4c3c-a311-3748913af72a-kube-api-access-8cm5p\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.948270 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948042    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-podres\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.948653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948614    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8cm5p\" (UniqueName: \"kubernetes.io/projected/c17947c1-bb33-4c3c-a311-3748913af72a-kube-api-access-8cm5p\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.948653 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948226    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-podres\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.948815 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948658    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-sys\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.948815 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948708    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-proc\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.948815 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-lib-modules\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.948815 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-proc\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.949031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-sys\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.949031 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.948897    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c17947c1-bb33-4c3c-a311-3748913af72a-lib-modules\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:39.959037 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:39.958498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8cm5p\" (UniqueName: \"kubernetes.io/projected/c17947c1-bb33-4c3c-a311-3748913af72a-kube-api-access-8cm5p\") pod \"perf-node-gather-daemonset-npc8w\" (UID: \"c17947c1-bb33-4c3c-a311-3748913af72a\") " pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:40.050475 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.050394    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:40.213848 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.213626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"]
Jul 28 17:52:40.354114 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.354072    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w" event={"ID":"c17947c1-bb33-4c3c-a311-3748913af72a","Type":"ContainerStarted","Data":"acab09954df0df0094ff55e31bb9174a79d878490875d4689d8318413ab30bb3"}
Jul 28 17:52:40.354308 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.354140    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:40.354308 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.354163    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w" event={"ID":"c17947c1-bb33-4c3c-a311-3748913af72a","Type":"ContainerStarted","Data":"54fd862dc71dba330f49c9c0d36bf7313378dabbd71eaec0946fe59349982fe2"}
Jul 28 17:52:40.371733 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.371658    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w" podStartSLOduration=1.371642622 podStartE2EDuration="1.371642622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 17:52:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 17:52:40.371132848 +0000 UTC m=+2137.304532268" watchObservedRunningTime="2026-07-28 17:52:40.371642622 +0000 UTC m=+2137.305042037"
Jul 28 17:52:40.640701 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.640639    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5sv95_c20a8a35-85b3-4c80-98f7-55055e247acb/dns/0.log"
Jul 28 17:52:40.664895 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.664868    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5sv95_c20a8a35-85b3-4c80-98f7-55055e247acb/kube-rbac-proxy/0.log"
Jul 28 17:52:40.850282 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:40.850248    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qh26v_9b96ddc0-b47c-4e45-989d-fcd1d1cd2e8c/dns-node-resolver/0.log"
Jul 28 17:52:41.325790 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:41.325762    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6b6qm_ccbf21fa-84d8-4348-bc89-36ea89e9284f/node-ca/0.log"
Jul 28 17:52:42.472907 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:42.472877    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-zm59x_272d8332-5d3c-4556-bcc6-57b9a2f9a946/serve-healthcheck-canary/0.log"
Jul 28 17:52:42.967934 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:42.967905    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qlxg9_53101351-b715-4d1c-9072-67c1165ede98/kube-rbac-proxy/0.log"
Jul 28 17:52:42.989214 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:42.989193    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qlxg9_53101351-b715-4d1c-9072-67c1165ede98/exporter/0.log"
Jul 28 17:52:43.010860 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:43.010843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qlxg9_53101351-b715-4d1c-9072-67c1165ede98/extractor/0.log"
Jul 28 17:52:45.257152 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:45.257109    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_model-serving-api-7b954ffc95-hr54x_e2c585ba-1b55-4306-a634-ca96d8490579/server/0.log"
Jul 28 17:52:45.393711 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:45.393647    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-f2899_ce51ebc8-0f9c-41ca-a52e-45a44f34928d/s3-init/0.log"
Jul 28 17:52:45.408539 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:45.408515    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-f2899_ce51ebc8-0f9c-41ca-a52e-45a44f34928d/download-hf-model/0.log"
Jul 28 17:52:45.437429 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:45.437399    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-lkngb_35278a5d-2d1b-419a-9004-4b2e8f9eb4ee/seaweedfs/0.log"
Jul 28 17:52:47.376344 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:47.376312    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cjjwk/perf-node-gather-daemonset-npc8w"
Jul 28 17:52:49.545423 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:49.545392    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-qc9gg_ff883792-b637-477e-a327-f0e44129405b/migrator/0.log"
Jul 28 17:52:49.569606 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:49.569567    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-6865c9967-qc9gg_ff883792-b637-477e-a327-f0e44129405b/graceful-termination/0.log"
Jul 28 17:52:51.003600 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.003573    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtzv7_a728637b-8a7c-4a0a-bb2d-b9b18c258162/kube-multus-additional-cni-plugins/0.log"
Jul 28 17:52:51.026011 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.025937    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtzv7_a728637b-8a7c-4a0a-bb2d-b9b18c258162/egress-router-binary-copy/0.log"
Jul 28 17:52:51.048416 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.048316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtzv7_a728637b-8a7c-4a0a-bb2d-b9b18c258162/cni-plugins/0.log"
Jul 28 17:52:51.069324 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.069299    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtzv7_a728637b-8a7c-4a0a-bb2d-b9b18c258162/bond-cni-plugin/0.log"
Jul 28 17:52:51.090919 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.090896    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtzv7_a728637b-8a7c-4a0a-bb2d-b9b18c258162/routeoverride-cni/0.log"
Jul 28 17:52:51.114761 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.114737    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtzv7_a728637b-8a7c-4a0a-bb2d-b9b18c258162/whereabouts-cni-bincopy/0.log"
Jul 28 17:52:51.136083 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.136061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-dtzv7_a728637b-8a7c-4a0a-bb2d-b9b18c258162/whereabouts-cni/0.log"
Jul 28 17:52:51.383951 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.383925    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-wh264_35770e90-7d87-4644-918f-aaea48f2e530/kube-multus/0.log"
Jul 28 17:52:51.563220 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.563193    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rlbl4_26a3cd7c-de8f-4208-b600-b598dd82afd4/network-metrics-daemon/0.log"
Jul 28 17:52:51.626771 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:51.626745    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rlbl4_26a3cd7c-de8f-4208-b600-b598dd82afd4/kube-rbac-proxy/0.log"
Jul 28 17:52:52.384220 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:52.384193    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56rvp_17c64fb3-f111-4c84-9b47-6c4695055d2f/ovn-controller/0.log"
Jul 28 17:52:52.402111 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:52.402090    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56rvp_17c64fb3-f111-4c84-9b47-6c4695055d2f/ovn-acl-logging/0.log"
Jul 28 17:52:52.424281 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:52.424245    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56rvp_17c64fb3-f111-4c84-9b47-6c4695055d2f/kube-rbac-proxy-node/0.log"
Jul 28 17:52:52.453814 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:52.453793    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56rvp_17c64fb3-f111-4c84-9b47-6c4695055d2f/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 17:52:52.482977 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:52.482957    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56rvp_17c64fb3-f111-4c84-9b47-6c4695055d2f/northd/0.log"
Jul 28 17:52:52.505725 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:52.505707    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56rvp_17c64fb3-f111-4c84-9b47-6c4695055d2f/nbdb/0.log"
Jul 28 17:52:52.531019 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:52.531004    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56rvp_17c64fb3-f111-4c84-9b47-6c4695055d2f/sbdb/0.log"
Jul 28 17:52:52.581098 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:52.581076    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-56rvp_17c64fb3-f111-4c84-9b47-6c4695055d2f/ovnkube-controller/0.log"
Jul 28 17:52:54.019757 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:54.019730    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-mlbvr_94714598-e34a-4953-b07d-c1b9037a561e/network-check-target-container/0.log"
Jul 28 17:52:54.921868 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:54.921843    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-z4k8t_79946742-16f6-4416-a9d2-e54871e8ed2d/iptables-alerter/0.log"
Jul 28 17:52:55.645388 ip-10-0-137-64 kubenswrapper[2563]: I0728 17:52:55.645312    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-sgtxd_57fa9c43-75d8-4b1b-91cc-b4fe41d43dc4/tuned/0.log"