Jul 21 17:25:38.435481 ip-10-0-129-22 systemd[1]: Starting Kubernetes Kubelet...
Jul 21 17:25:38.774928 ip-10-0-129-22 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:25:38.774928 ip-10-0-129-22 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 21 17:25:38.774928 ip-10-0-129-22 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:25:38.774928 ip-10-0-129-22 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 21 17:25:38.775609 ip-10-0-129-22 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 21 17:25:38.775926 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.775816    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 21 17:25:38.782511 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782487    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:38.782511 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782506    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:38.782511 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782510    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:38.782511 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782514    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:38.782511 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782516    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:38.782511 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782519    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782522    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782525    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782528    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782530    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782533    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782535    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782538    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782540    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782543    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782545    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782548    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782550    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782552    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782554    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782557    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782559    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782562    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782564    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:38.782740 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782574    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782577    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782579    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782582    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782584    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782587    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782589    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782592    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782594    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782597    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782600    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782602    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782604    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782607    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782609    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782612    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782615    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782618    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782621    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782623    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:38.783199 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782626    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782628    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782631    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782633    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782636    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782638    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782640    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782642    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782645    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782647    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782650    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782652    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782655    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782657    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782660    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782662    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782664    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782667    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782671    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:38.783697 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782675    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782678    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782681    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782685    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782687    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782690    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782693    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782698    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782701    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782704    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782707    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782709    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782712    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782714    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782717    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782720    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782722    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782726    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782729    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:38.784151 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782731    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782735    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782737    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782740    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782743    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782745    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782748    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782752    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782754    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782757    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782760    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782762    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782764    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782767    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782770    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782772    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.782775    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783504    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783513    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783517    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:38.784650 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783520    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783523    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783526    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783530    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783532    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783535    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783538    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783541    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783543    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783546    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783549    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783551    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783554    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783556    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783559    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783562    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783564    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783566    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783569    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783571    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:38.785130 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783573    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783576    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783578    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783581    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783583    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783585    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783588    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783592    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783594    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783597    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783599    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783602    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783605    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783607    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783609    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783612    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783614    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783617    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783619    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:38.785633 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783621    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783624    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783626    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783628    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783630    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783633    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783635    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783637    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783641    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783643    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783645    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783648    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783650    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783652    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783655    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783657    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783660    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783662    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783665    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783667    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:38.786081 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783670    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783673    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783675    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783678    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783682    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783686    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783688    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783691    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783693    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783696    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783698    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783701    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783703    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783706    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783708    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783711    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783713    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783716    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783718    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783721    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:38.786587 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783724    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783727    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783730    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783732    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783735    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783737    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783740    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783742    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783745    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783748    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783750    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783752    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783755    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783757    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783760    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783762    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.783766    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784141    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784147    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:38.787085 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784150    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784153    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784156    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784158    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784161    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784164    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784167    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784169    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784173    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784176    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784179    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784182    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784185    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784188    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784191    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784194    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784197    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784199    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784202    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:38.787585 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784204    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784206    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784209    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784211    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784231    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784234    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784237    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784240    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784243    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784246    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784248    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784251    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784253    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784256    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784258    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784261    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784263    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784266    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784268    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784271    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:38.788044 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784273    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784276    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784278    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784281    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784283    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784286    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784288    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784291    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784294    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784296    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784299    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784302    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784304    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784307    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784309    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784312    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784314    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784317    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784319    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784322    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:38.788546 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784325    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784328    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784330    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784332    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784335    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784337    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784340    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784342    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784345    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784347    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784349    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784351    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784354    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784356    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784359    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784361    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784363    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784366    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784369    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784371    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:38.789082 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784374    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784377    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784380    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784382    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784385    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784387    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784389    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784393    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784396    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784399    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784404    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784407    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784410    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784413    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784416    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784419    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784421    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.784424    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784486    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784493    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 21 17:25:38.789582 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784499    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784503    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784507    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784510    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784514    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784518    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784521    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784524    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784527    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784530    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784536    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784539    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784542    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784544    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784548    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784550    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784554    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784557    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784560    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784563    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784566    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784569    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784572    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 21 17:25:38.790059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784575    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784578    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784581    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784584    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784586    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784589    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784592    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784595    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784598    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784601    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784604    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784606    2563 flags.go:64] FLAG: --enable-server="true"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784609    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784613    2563 flags.go:64] FLAG: --event-burst="100"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784616    2563 flags.go:64] FLAG: --event-qps="50"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784619    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784622    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784624    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784628    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784633    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784636    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784640    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784642    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784645    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784648    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 21 17:25:38.790666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784651    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784654    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784657    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784660    2563 flags.go:64] FLAG: --feature-gates=""
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784664    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784667    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784670    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784673    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784675    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784678    2563 flags.go:64] FLAG: --help="false"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784680    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-129-22.ec2.internal"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784684    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784686    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784689    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784692    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784695    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784698    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784701    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784703    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784706    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784708    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784711    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784714    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784716    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 21 17:25:38.791330 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784719    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784722    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784727    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784730    2563 flags.go:64] FLAG: --lock-file=""
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784732    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784735    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784738    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784743    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784747    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784749    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784752    2563 flags.go:64] FLAG: --logging-format="text"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784755    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784758    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784760    2563 flags.go:64] FLAG: --manifest-url=""
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784763    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784767    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784770    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784774    2563 flags.go:64] FLAG: --max-pods="110"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784777    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784779    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784782    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784785    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784788    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784791    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784793    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 21 17:25:38.791913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784801    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784804    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784806    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784809    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784812    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784817    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784819    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784822    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784825    2563 flags.go:64] FLAG: --port="10250"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784829    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784834    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09428c29f5f8575c0"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784837    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784840    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784842    2563 flags.go:64] FLAG: --register-node="true"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784845    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784848    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784852    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784854    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784857    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784861    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784864    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784867    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784869    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784872    2563 flags.go:64] FLAG: --runonce="false"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784875    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 21 17:25:38.792539 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784878    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784880    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784883    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784886    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784888    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784891    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784894    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784896    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784899    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784902    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784904    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784907    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784910    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784912    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784917    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784920    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784923    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784928    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784931    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784933    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784936    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784939    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784941    2563 flags.go:64] FLAG: --v="2"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784945    2563 flags.go:64] FLAG: --version="false"
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784955    2563 flags.go:64] FLAG: --vmodule=""
Jul 21 17:25:38.793151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784959    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.784962    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785076    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785079    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785082    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785085    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785088    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785090    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785093    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785095    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785097    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785100    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785103    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785106    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785108    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785110    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785113    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785115    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785118    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785120    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:38.793794 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785123    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785125    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785127    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785131    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785135    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785138    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785146    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785149    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785152    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785155    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785158    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785160    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785164    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785167    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785170    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785173    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785175    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785178    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785181    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:38.794361 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785184    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785186    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785188    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785191    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785193    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785195    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785198    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785200    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785203    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785206    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785209    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785212    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785229    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785232    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785235    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785237    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785241    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785244    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785247    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:38.794838 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785250    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785252    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785255    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785257    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785260    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785262    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785265    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785268    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785271    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785273    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785276    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785278    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785281    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785283    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785286    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785289    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785291    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785293    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785296    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:38.795310 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785299    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785303    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785306    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785308    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785311    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785313    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785315    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785318    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785320    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785323    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785326    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785329    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785332    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785334    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785337    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785339    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785342    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785344    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785347    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785349    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:38.795780 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785352    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785354    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785357    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.785360    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.785365    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.792771    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.792786    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.793317    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.793324    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793398    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793403    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793407    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793410    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793413    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793416    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:38.796272 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793418    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793421    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793423    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793426    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793428    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793431    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793434    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793436    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793439    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793441    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793444    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793446    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793449    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793451    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793454    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793458    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793462    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793464    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793467    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:38.796713 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793470    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793472    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793475    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793478    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793481    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793485    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793489    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793492    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793495    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793498    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793501    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793504    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793506    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793509    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793511    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793514    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793516    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793519    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793521    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793524    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:38.797160 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793526    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793529    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793531    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793534    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793536    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793539    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793541    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793544    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793546    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793548    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793551    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793553    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793559    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793562    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793564    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793567    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793570    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793573    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793575    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793578    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:38.797756 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793581    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793583    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793586    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793588    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793591    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793593    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793596    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793598    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793601    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793603    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793606    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793608    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793611    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793614    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793616    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793619    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793621    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793623    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793626    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:38.798281 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793628    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793632    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793635    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793638    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793641    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793644    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793647    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793650    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793653    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793656    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793659    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793661    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793663    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793666    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:38.798940 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793669    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.793674    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793814    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793819    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793822    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793824    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793827    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793829    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793832    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793835    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793837    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793840    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793844    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793847    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793850    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 21 17:25:38.799383 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793852    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793855    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793857    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793861    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793864    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793867    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793876    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793879    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793882    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793885    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793894    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793897    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793900    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793903    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793905    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793908    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793910    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793912    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793915    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 21 17:25:38.800006 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793918    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793920    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793923    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793925    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793928    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793930    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793933    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793935    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793937    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793940    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793942    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793945    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793947    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793949    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793952    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793954    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793956    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793959    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793961    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793964    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 21 17:25:38.800744 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793966    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793968    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793971    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793974    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793982    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793985    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793987    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793990    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793992    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793995    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.793997    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794000    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794002    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794005    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794007    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794009    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794012    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794014    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794017    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794019    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 21 17:25:38.801304 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794021    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794024    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794026    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794029    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794031    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794033    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794036    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794038    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794040    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794043    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794045    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794048    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794050    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794052    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794055    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794058    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794061    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794069    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794071    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794074    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 21 17:25:38.802107 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794077    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794079    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794082    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794084    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794086    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794089    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:38.794091    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.794096    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.794193    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.796326    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.797484    2563 server.go:1019] "Starting client certificate rotation"
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.797615    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.797665    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 21 17:25:38.802696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.799799    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 17:25:38.803022 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.802256    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 21 17:25:38.813308 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.813287    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 21 17:25:38.817983 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.817968    2563 log.go:25] "Validated CRI v1 image API"
Jul 21 17:25:38.818805 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.818792    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 21 17:25:38.822162 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.822134    2563 fs.go:135] Filesystem UUIDs: map[507153fb-4cb6-43bc-bda4-d5455c4df764:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 df4681d5-3ebd-4b88-a408-bb84175a5dde:/dev/nvme0n1p3]
Jul 21 17:25:38.822247 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.822162    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 21 17:25:38.826997 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.826887    2563 manager.go:216] Machine: {Timestamp:2026-07-21 17:25:38.825795338 +0000 UTC m=+0.288373124 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102724 MemoryCapacity:32812175360 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2e6a2bdb24e84f5cebd058fbc6c336 SystemUUID:ec2e6a2b-db24-e84f-5ceb-d058fbc6c336 BootID:05dbf506-258b-4318-a8ec-400a5ca1a339 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406089728 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406085632 Type:vfs Inodes:4005392 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:26:17:b0:c6:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:26:17:b0:c6:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c6:04:72:12:b2:1a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812175360 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 21 17:25:38.826997 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.826996    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 21 17:25:38.827101 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.827071    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 21 17:25:38.827414 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.827400    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 17:25:38.829704 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.829680    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 21 17:25:38.829837 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.829707    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-22.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 21 17:25:38.830413 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.830403    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 21 17:25:38.830451 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.830415    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 21 17:25:38.830451 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.830427    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 17:25:38.831160 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.831150    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 21 17:25:38.832740 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.832730    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 21 17:25:38.835845 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.835834    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 17:25:38.835953 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.835944    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 21 17:25:38.835990 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.835977    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 21 17:25:38.835990 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.835987    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 21 17:25:38.836046 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.835997    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 21 17:25:38.836046 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.836005    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 21 17:25:38.836046 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.836014    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 21 17:25:38.836388 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.836377    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 17:25:38.836440 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.836396    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 21 17:25:38.838429 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.838415    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 21 17:25:38.840157 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.840142    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 21 17:25:38.840233 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.840173    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 21 17:25:38.840905 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.840894    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 21 17:25:38.840947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.840910    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 21 17:25:38.840947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.840916    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 21 17:25:38.840947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.840921    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 21 17:25:38.840947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.840929    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 21 17:25:38.840947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.840934    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 21 17:25:38.841117 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.841109    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 21 17:25:38.841149 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.841118    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 21 17:25:38.841149 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.841124    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 21 17:25:38.841149 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.841130    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 21 17:25:38.841149 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.841145    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 21 17:25:38.841272 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.841153    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 21 17:25:38.842925 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.842914    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 21 17:25:38.842964 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.842927    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 21 17:25:38.846261 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.846243    2563 server.go:1268] "Started kubelet"
Jul 21 17:25:38.846384 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.846343    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 21 17:25:38.846501 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.846387    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 21 17:25:38.846562 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.846502    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 21 17:25:38.847233 ip-10-0-129-22 systemd[1]: Started Kubernetes Kubelet.
Jul 21 17:25:38.847882 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.847848    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 21 17:25:38.851877 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.851852    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 21 17:25:38.851961 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.851946    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 21 17:25:38.852424 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.852405    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-22.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 21 17:25:38.853154 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.853116    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 21 17:25:38.853800 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.853779    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 21 17:25:38.855600 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.853887    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-22.ec2.internal.18c45e078ba50f10  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-22.ec2.internal,UID:ip-10-0-129-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-22.ec2.internal,},FirstTimestamp:2026-07-21 17:25:38.846191376 +0000 UTC m=+0.308769162,LastTimestamp:2026-07-21 17:25:38.846191376 +0000 UTC m=+0.308769162,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-22.ec2.internal,}"
Jul 21 17:25:38.858554 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.858403    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n7vhg"
Jul 21 17:25:38.859320 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.859305    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 21 17:25:38.859390 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.859356    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 21 17:25:38.859438 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.859394    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:38.859978 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.859960    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 21 17:25:38.860082 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860066    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 21 17:25:38.860147 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860084    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 21 17:25:38.860200 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860171    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 21 17:25:38.860200 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860181    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 21 17:25:38.860480 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860465    2563 factory.go:153] Registering CRI-O factory
Jul 21 17:25:38.860559 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860485    2563 factory.go:223] Registration of the crio container factory successfully
Jul 21 17:25:38.860559 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860540    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 21 17:25:38.860559 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860549    2563 factory.go:55] Registering systemd factory
Jul 21 17:25:38.860559 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860556    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 21 17:25:38.860736 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860577    2563 factory.go:103] Registering Raw factory
Jul 21 17:25:38.860736 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860591    2563 manager.go:1196] Started watching for new ooms in manager
Jul 21 17:25:38.860991 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.860974    2563 manager.go:349] Starting recovery of all containers
Jul 21 17:25:38.864638 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.864612    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-n7vhg"
Jul 21 17:25:38.866400 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.866366    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 21 17:25:38.866492 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.866380    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-22.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 21 17:25:38.870487 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.870460    2563 manager.go:354] Recovery completed
Jul 21 17:25:38.874543 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.874527    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:38.876394 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.876377    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:38.876469 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.876407    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:38.876469 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.876422    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:38.876968 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.876954    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 21 17:25:38.877025 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.876969    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 21 17:25:38.877025 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.876988    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 21 17:25:38.877025 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.876955    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:38.877803 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.877788    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:38.877879 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.877819    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:38.877879 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.877833    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:38.878199 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.878188    2563 policy_none.go:49] "None policy: Start"
Jul 21 17:25:38.878269 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.878203    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 21 17:25:38.878269 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.878228    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 21 17:25:38.878516 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.878438    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-22.ec2.internal.18c45e078d71e60b  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-22.ec2.internal,UID:ip-10-0-129-22.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-22.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-22.ec2.internal,},FirstTimestamp:2026-07-21 17:25:38.876392971 +0000 UTC m=+0.338970760,LastTimestamp:2026-07-21 17:25:38.876392971 +0000 UTC m=+0.338970760,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-22.ec2.internal,}"
Jul 21 17:25:38.879740 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.879729    2563 policy_none.go:47] "Start"
Jul 21 17:25:38.913869 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.913742    2563 manager.go:337] "Starting Device Plugin manager"
Jul 21 17:25:38.913961 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.913893    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 21 17:25:38.913961 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.913906    2563 server.go:87] "Starting device plugin registration server"
Jul 21 17:25:38.914146 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.914133    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 21 17:25:38.914208 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.914148    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 21 17:25:38.914303 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.914285    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 21 17:25:38.914423 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.914377    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 21 17:25:38.914423 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.914387    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 21 17:25:38.915130 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.915111    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 21 17:25:38.915205 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.915158    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:38.996544 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.996514    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 21 17:25:38.997848 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.997830    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 21 17:25:38.997894 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.997877    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 21 17:25:38.997926 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:38.997902    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 21 17:25:38.997962 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:38.997943    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 21 17:25:39.000285 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.000266    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:39.015127 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.015109    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:39.015920 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.015905    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:39.015984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.015933    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:39.015984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.015943    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:39.015984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.015964    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.023703 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.023684    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.023780 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.023706    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-22.ec2.internal\": node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.046744 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.046696    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.098594 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.098570    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"]
Jul 21 17:25:39.098675 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.098621    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:39.099391 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.099376    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:39.099468 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.099406    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:39.099468 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.099420    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:39.100710 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.100695    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:39.100809 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.100795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.100854 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.100845    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:39.101383 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.101368    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:39.101445 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.101395    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:39.101445 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.101405    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:39.101513 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.101369    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:39.101513 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.101472    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:39.101513 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.101484    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:39.102685 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.102670    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.102757 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.102696    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 21 17:25:39.103321 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.103304    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientMemory"
Jul 21 17:25:39.103401 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.103333    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasNoDiskPressure"
Jul 21 17:25:39.103401 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.103342    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeHasSufficientPID"
Jul 21 17:25:39.136819 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.136800    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-22.ec2.internal\" not found" node="ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.141325 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.141310    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-22.ec2.internal\" not found" node="ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.147302 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.147288    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.161577 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.161558    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c4726a2cf6e4377be6ec06d1c802bab6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-22.ec2.internal\" (UID: \"c4726a2cf6e4377be6ec06d1c802bab6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.161651 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.161580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10fe471e90fa9c7c931061a10f11dce9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal\" (UID: \"10fe471e90fa9c7c931061a10f11dce9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.161651 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.161598    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10fe471e90fa9c7c931061a10f11dce9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal\" (UID: \"10fe471e90fa9c7c931061a10f11dce9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.247868 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.247847    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.262228 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.262191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c4726a2cf6e4377be6ec06d1c802bab6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-22.ec2.internal\" (UID: \"c4726a2cf6e4377be6ec06d1c802bab6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.262306 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.262232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10fe471e90fa9c7c931061a10f11dce9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal\" (UID: \"10fe471e90fa9c7c931061a10f11dce9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.262306 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.262248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10fe471e90fa9c7c931061a10f11dce9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal\" (UID: \"10fe471e90fa9c7c931061a10f11dce9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.262306 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.262293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/10fe471e90fa9c7c931061a10f11dce9-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal\" (UID: \"10fe471e90fa9c7c931061a10f11dce9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.262419 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.262311    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/c4726a2cf6e4377be6ec06d1c802bab6-config\") pod \"kube-apiserver-proxy-ip-10-0-129-22.ec2.internal\" (UID: \"c4726a2cf6e4377be6ec06d1c802bab6\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.262419 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.262331    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/10fe471e90fa9c7c931061a10f11dce9-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal\" (UID: \"10fe471e90fa9c7c931061a10f11dce9\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.348619 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.348598    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.439213 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.439173    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.445631 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.445608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:39.448695 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.448676    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.549619 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.549587    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.650211 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.650143    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.750645 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.750614    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.797167 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.797141    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 21 17:25:39.797562 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.797299    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 21 17:25:39.851655 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:39.851630    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:39.872083 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.872051    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-20 17:20:40 +0000 UTC" deadline="2028-02-24 11:16:42.587834222 +0000 UTC"
Jul 21 17:25:39.872083 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.872080    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13985h51m2.71575709s"
Jul 21 17:25:39.876240 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:39.876206    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:41.830892 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:41.830862    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4726a2cf6e4377be6ec06d1c802bab6.slice/crio-ba147f105cae68daec86ca5ab1a6587ea769f75b3aa4a2c3bf2a61c8c08974b3 WatchSource:0}: Error finding container ba147f105cae68daec86ca5ab1a6587ea769f75b3aa4a2c3bf2a61c8c08974b3: Status 404 returned error can't find the container with id ba147f105cae68daec86ca5ab1a6587ea769f75b3aa4a2c3bf2a61c8c08974b3
Jul 21 17:25:41.831187 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:41.831164    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10fe471e90fa9c7c931061a10f11dce9.slice/crio-d7d04d19c93d661d5a99880ce9170c99cbca0bd15bd42bbac755ce8a508df236 WatchSource:0}: Error finding container d7d04d19c93d661d5a99880ce9170c99cbca0bd15bd42bbac755ce8a508df236: Status 404 returned error can't find the container with id d7d04d19c93d661d5a99880ce9170c99cbca0bd15bd42bbac755ce8a508df236
Jul 21 17:25:41.835879 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:41.835859    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:25:41.856278 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:41.856252    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:41.905743 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:41.905702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal" event={"ID":"10fe471e90fa9c7c931061a10f11dce9","Type":"ContainerStarted","Data":"d7d04d19c93d661d5a99880ce9170c99cbca0bd15bd42bbac755ce8a508df236"}
Jul 21 17:25:41.906677 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:41.906656    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal" event={"ID":"c4726a2cf6e4377be6ec06d1c802bab6","Type":"ContainerStarted","Data":"ba147f105cae68daec86ca5ab1a6587ea769f75b3aa4a2c3bf2a61c8c08974b3"}
Jul 21 17:25:41.956982 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:41.956959    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:41.961199 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:41.961184    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:42.057972 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.057919    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:42.158429 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.158409    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:42.259210 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.259179    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-22.ec2.internal\" not found"
Jul 21 17:25:42.331672 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.331610    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:42.364845 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.364653    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:42.374938 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.374913    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 17:25:42.375771 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.375750    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal"
Jul 21 17:25:42.387163 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.387139    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 21 17:25:42.742018 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.741947    2563 apiserver.go:52] "Watching apiserver"
Jul 21 17:25:42.755154 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.755001    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 21 17:25:42.756151 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.756123    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-tl99f","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal","openshift-multus/multus-additional-cni-plugins-5xzbr","openshift-multus/multus-lgh5k","openshift-network-operator/iptables-alerter-5wmkn","kube-system/global-pull-secret-syncer-p8lvn","kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c","openshift-dns/node-resolver-5njds","openshift-image-registry/node-ca-p7867","openshift-multus/network-metrics-daemon-rz8hm","openshift-network-diagnostics/network-check-target-qq9ln","openshift-ovn-kubernetes/ovnkube-node-shlvf","kube-system/konnectivity-agent-v89wr"]
Jul 21 17:25:42.757751 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.757728    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.759543 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.759407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.761939 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.760576    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 21 17:25:42.761939 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.760939    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 21 17:25:42.761939 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.761125    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 21 17:25:42.761939 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.761128    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-k8plg\""
Jul 21 17:25:42.761939 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.761573    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.762779 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.762757    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 21 17:25:42.763019 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.763004    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 21 17:25:42.763197 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.763179    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 21 17:25:42.764088 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.763637    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 21 17:25:42.764088 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.763775    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 21 17:25:42.764088 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.763854    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 21 17:25:42.764088 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.764039    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-f22ww\""
Jul 21 17:25:42.764331 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.764220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 21 17:25:42.764994 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.764967    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 21 17:25:42.767624 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.766528    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 21 17:25:42.767624 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.766559    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 21 17:25:42.767624 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.766760    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 21 17:25:42.767624 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.766962    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z7jvt\""
Jul 21 17:25:42.767624 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.767142    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 21 17:25:42.769109 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.768834    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:42.769109 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.768923    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:42.770667 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.770604    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:42.770857 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.770819    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:42.772220 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.772200    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.773895 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.773867    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.774864 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.774599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.774864 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.774831    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kxl5w\""
Jul 21 17:25:42.775278 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.775118    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 21 17:25:42.775278 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.775248    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:25:42.776364 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776343    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 21 17:25:42.776475 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776459    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 21 17:25:42.776564 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 21 17:25:42.776643 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776630    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-xpfvp\""
Jul 21 17:25:42.776778 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776763    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776771    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.776858    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-cni-bin\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-os-release\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.776997    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-env-overrides\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777062    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-etc-selinux\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777118    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-etc-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777118    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777144    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-log-socket\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777175    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cnibin\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777255    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-socket-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.778281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-device-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777321    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-slash\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777369    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-systemd\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777425    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-var-lib-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovnkube-script-lib\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777459    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-cni-netd\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-system-cni-dir\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf4tv\" (UniqueName: \"kubernetes.io/projected/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-kube-api-access-rf4tv\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777530    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-655g9\""
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-registration-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777550    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-node-log\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777580    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777612    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777660    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777687    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovnkube-config\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cv8bt\" (UniqueName: \"kubernetes.io/projected/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-kube-api-access-cv8bt\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.778996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cpw9b\" (UniqueName: \"kubernetes.io/projected/9af59be9-5692-410e-a7b7-e15daa138583-kube-api-access-cpw9b\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.779781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-systemd-units\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.779781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovn-node-metrics-cert\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.779781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777855    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-sys-fs\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.779781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777872    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 21 17:25:42.779781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777878    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-kubelet\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.779781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777903    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-run-netns\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.779781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.777924    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-ovn\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.779781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.778069    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 21 17:25:42.785223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.781241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.785223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.784195    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 21 17:25:42.785223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.784497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-6p9l8\""
Jul 21 17:25:42.785223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.784723    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 21 17:25:42.785223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.785019    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 21 17:25:42.789132 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.788860    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.790557 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.790535    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:25:42.791268 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.791252    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 21 17:25:42.791453 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.791435    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-ng26v\""
Jul 21 17:25:42.793028 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.792967    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 21 17:25:42.793371 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.793225    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 21 17:25:42.793371 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.793227    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-8vqt7\""
Jul 21 17:25:42.799118 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.799100    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xv4hn"
Jul 21 17:25:42.805821 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.805784    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xv4hn"
Jul 21 17:25:42.865626 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.865601    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 21 17:25:42.878599 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878569    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-netns\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.878747 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878619    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-multus-certs\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.878747 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-systemd-units\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.878747 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878676    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-sys-fs\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.878747 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878699    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysctl-d\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.878747 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13d2df84-0465-42b2-9ff8-cb39aaa95e33-dbus\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.878977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5r99\" (UniqueName: \"kubernetes.io/projected/ba892592-38ce-46ab-8073-907a0e715c39-kube-api-access-j5r99\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.878977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7573a048-9229-4379-812a-1f7e92b6a852-konnectivity-ca\") pod \"konnectivity-agent-v89wr\" (UID: \"7573a048-9229-4379-812a-1f7e92b6a852\") " pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:25:42.878977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878833    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-kubelet\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.878977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878855    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-ovn\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.878977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878878    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-os-release\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.878977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.878977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-run\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.878977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878944    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vk9c\" (UniqueName: \"kubernetes.io/projected/a4b01391-be0e-4c48-a84d-2842bedb9248-kube-api-access-9vk9c\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.878978    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879010    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-k8s-cni-cncf-io\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-etc-selinux\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879081    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-systemd-units\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879095    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-kubernetes\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba892592-38ce-46ab-8073-907a0e715c39-serviceca\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-log-socket\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-sys-fs\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-device-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879199    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysconfig\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879233    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-kubelet\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.879255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysctl-conf\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879266    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b01391-be0e-4c48-a84d-2842bedb9248-tmp\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879292    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6ad5b4-7a10-49b2-9618-440018cb51c3-cni-binary-copy\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879317    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-daemon-config\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovnkube-script-lib\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879359    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-host\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879383    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-cni-netd\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-etc-selinux\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-system-cni-dir\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf4tv\" (UniqueName: \"kubernetes.io/projected/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-kube-api-access-rf4tv\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-log-socket\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-registration-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879485    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7grn7\" (UniqueName: \"kubernetes.io/projected/b64a4088-eec0-4e59-bb26-7b4297a68fdf-kube-api-access-7grn7\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879498    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-device-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879509    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-system-cni-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879532    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7573a048-9229-4379-812a-1f7e92b6a852-agent-certs\") pod \"konnectivity-agent-v89wr\" (UID: \"7573a048-9229-4379-812a-1f7e92b6a852\") " pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:25:42.879689 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879554    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-ovn\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879582    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b64a4088-eec0-4e59-bb26-7b4297a68fdf-iptables-alerter-script\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-os-release\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-cni-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-etc-kubernetes\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovnkube-config\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-lib-modules\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879782    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-tuned\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879824    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13d2df84-0465-42b2-9ff8-cb39aaa95e33-kubelet-config\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879845    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-hostroot\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovn-node-metrics-cert\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879892    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-systemd\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879916    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qksvq\" (UniqueName: \"kubernetes.io/projected/27821ce6-26d8-4a08-b587-91eeac050017-kube-api-access-qksvq\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879941    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-run-netns\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879967    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.879990    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-cni-bin\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27821ce6-26d8-4a08-b587-91eeac050017-hosts-file\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880034    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-env-overrides\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880058    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880112    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-modprobe-d\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880137    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-kubelet\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880161    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-conf-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880192    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72g6q\" (UniqueName: \"kubernetes.io/projected/fe6ad5b4-7a10-49b2-9618-440018cb51c3-kube-api-access-72g6q\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-etc-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cnibin\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovnkube-script-lib\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-kubelet-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-socket-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880378    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b64a4088-eec0-4e59-bb26-7b4297a68fdf-host-slash\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880401    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-cni-bin\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.880868 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880427    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-cni-multus\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba892592-38ce-46ab-8073-907a0e715c39-host\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-tuning-conf-dir\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880483    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-slash\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880508    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-systemd\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-var-lib-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cnibin\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-cni-netd\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-run-systemd\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880733    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-system-cni-dir\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.880963    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-env-overrides\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881022    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-registration-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881029    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-socket-dir-parent\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881096    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c74gc\" (UniqueName: \"kubernetes.io/projected/8cfe68ed-2ecc-4369-adcd-235f6b199b24-kube-api-access-c74gc\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881118    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-node-log\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:42.881636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-sys\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881177    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27821ce6-26d8-4a08-b587-91eeac050017-tmp-dir\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881196    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-cnibin\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881212    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cni-binary-copy\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cv8bt\" (UniqueName: \"kubernetes.io/projected/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-kube-api-access-cv8bt\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881250    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cpw9b\" (UniqueName: \"kubernetes.io/projected/9af59be9-5692-410e-a7b7-e15daa138583-kube-api-access-cpw9b\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881277    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-var-lib-kubelet\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881286    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-var-lib-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881297    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-os-release\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881371    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-node-log\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881487    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/9af59be9-5692-410e-a7b7-e15daa138583-socket-dir\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881839    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-slash\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-run-ovn-kubernetes\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-etc-openvswitch\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882920 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881903    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-cni-bin\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882920 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-host-run-netns\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.882920 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.881985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovnkube-config\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.884620 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.884595    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-ovn-node-metrics-cert\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.886478 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.886457    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:42.886571 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.886483    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:42.886571 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.886496    2563 projected.go:196] Error preparing data for projected volume kube-api-access-7qzhf for pod openshift-network-diagnostics/network-check-target-qq9ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:42.886571 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.886555    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf podName:ff525c04-45a3-46cd-883e-fbbc3416f73c nodeName:}" failed. No retries permitted until 2026-07-21 17:25:43.386540081 +0000 UTC m=+2.944547165 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-7qzhf" (UniqueName: "kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf") pod "network-check-target-qq9ln" (UID: "ff525c04-45a3-46cd-883e-fbbc3416f73c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:42.888906 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.888887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf4tv\" (UniqueName: \"kubernetes.io/projected/77eb9156-b7f3-4f07-88f2-c5a2329e7bc4-kube-api-access-rf4tv\") pod \"multus-additional-cni-plugins-5xzbr\" (UID: \"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4\") " pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:42.889471 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.889453    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cv8bt\" (UniqueName: \"kubernetes.io/projected/3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e-kube-api-access-cv8bt\") pod \"ovnkube-node-shlvf\" (UID: \"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e\") " pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:42.889563 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.889548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cpw9b\" (UniqueName: \"kubernetes.io/projected/9af59be9-5692-410e-a7b7-e15daa138583-kube-api-access-cpw9b\") pod \"aws-ebs-csi-driver-node-clh7c\" (UID: \"9af59be9-5692-410e-a7b7-e15daa138583\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:42.942291 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.942265    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 21 17:25:42.982221 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.982192    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-run\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.982334 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.982238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vk9c\" (UniqueName: \"kubernetes.io/projected/a4b01391-be0e-4c48-a84d-2842bedb9248-kube-api-access-9vk9c\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.982334 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.982268    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.982744    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.982757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-run\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.982895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-k8s-cni-cncf-io\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.982948    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-kubernetes\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.982996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba892592-38ce-46ab-8073-907a0e715c39-serviceca\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-kubernetes\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-k8s-cni-cncf-io\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983149    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysconfig\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysconfig\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983351 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.983320    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret podName:13d2df84-0465-42b2-9ff8-cb39aaa95e33 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:43.483298053 +0000 UTC m=+3.041305158 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret") pod "global-pull-secret-syncer-p8lvn" (UID: "13d2df84-0465-42b2-9ff8-cb39aaa95e33") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysctl-conf\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysctl-conf\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b01391-be0e-4c48-a84d-2842bedb9248-tmp\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983493    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6ad5b4-7a10-49b2-9618-440018cb51c3-cni-binary-copy\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983644    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ba892592-38ce-46ab-8073-907a0e715c39-serviceca\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983700    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-daemon-config\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-host\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7grn7\" (UniqueName: \"kubernetes.io/projected/b64a4088-eec0-4e59-bb26-7b4297a68fdf-kube-api-access-7grn7\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-system-cni-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.983902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7573a048-9229-4379-812a-1f7e92b6a852-agent-certs\") pod \"konnectivity-agent-v89wr\" (UID: \"7573a048-9229-4379-812a-1f7e92b6a852\") " pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b64a4088-eec0-4e59-bb26-7b4297a68fdf-iptables-alerter-script\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983955    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-cni-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.983983    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-etc-kubernetes\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-lib-modules\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/fe6ad5b4-7a10-49b2-9618-440018cb51c3-cni-binary-copy\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984057    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-tuned\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13d2df84-0465-42b2-9ff8-cb39aaa95e33-kubelet-config\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984195    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-hostroot\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984200    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-etc-kubernetes\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-systemd\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984275    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qksvq\" (UniqueName: \"kubernetes.io/projected/27821ce6-26d8-4a08-b587-91eeac050017-kube-api-access-qksvq\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984307    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27821ce6-26d8-4a08-b587-91eeac050017-hosts-file\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984339    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-modprobe-d\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.984369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984369    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-kubelet\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984359    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-lib-modules\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-conf-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72g6q\" (UniqueName: \"kubernetes.io/projected/fe6ad5b4-7a10-49b2-9618-440018cb51c3-kube-api-access-72g6q\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984500    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-kubelet\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984536    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b64a4088-eec0-4e59-bb26-7b4297a68fdf-host-slash\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-cni-bin\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-cni-multus\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984628    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba892592-38ce-46ab-8073-907a0e715c39-host\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984664    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-socket-dir-parent\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984693    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c74gc\" (UniqueName: \"kubernetes.io/projected/8cfe68ed-2ecc-4369-adcd-235f6b199b24-kube-api-access-c74gc\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984707    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/13d2df84-0465-42b2-9ff8-cb39aaa95e33-kubelet-config\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-sys\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984773    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-conf-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984781    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27821ce6-26d8-4a08-b587-91eeac050017-tmp-dir\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984842    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-cnibin\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984875    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984908    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-var-lib-kubelet\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.984977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-os-release\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-netns\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-multus-certs\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/27821ce6-26d8-4a08-b587-91eeac050017-tmp-dir\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985130    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysctl-d\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985165    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13d2df84-0465-42b2-9ff8-cb39aaa95e33-dbus\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985212    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5r99\" (UniqueName: \"kubernetes.io/projected/ba892592-38ce-46ab-8073-907a0e715c39-kube-api-access-j5r99\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7573a048-9229-4379-812a-1f7e92b6a852-konnectivity-ca\") pod \"konnectivity-agent-v89wr\" (UID: \"7573a048-9229-4379-812a-1f7e92b6a852\") " pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-netns\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.985729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985663    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b64a4088-eec0-4e59-bb26-7b4297a68fdf-iptables-alerter-script\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/7573a048-9229-4379-812a-1f7e92b6a852-konnectivity-ca\") pod \"konnectivity-agent-v89wr\" (UID: \"7573a048-9229-4379-812a-1f7e92b6a852\") " pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.985964    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-cnibin\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-cni-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.986058    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986080    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-systemd\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.984244    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-hostroot\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:42.986111    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs podName:8cfe68ed-2ecc-4369-adcd-235f6b199b24 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:43.486095903 +0000 UTC m=+3.044102989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs") pod "network-metrics-daemon-rz8hm" (UID: "8cfe68ed-2ecc-4369-adcd-235f6b199b24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986175    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-var-lib-kubelet\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.986182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/27821ce6-26d8-4a08-b587-91eeac050017-hosts-file\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986298    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-modprobe-d\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a4b01391-be0e-4c48-a84d-2842bedb9248-tmp\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-os-release\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986458    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-sysctl-d\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986506    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-run-multus-certs\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ba892592-38ce-46ab-8073-907a0e715c39-host\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986566    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-host\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986575    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-cni-bin\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986589    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-socket-dir-parent\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.986655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986631    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-host-var-lib-cni-multus\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.987223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986843    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/7573a048-9229-4379-812a-1f7e92b6a852-agent-certs\") pod \"konnectivity-agent-v89wr\" (UID: \"7573a048-9229-4379-812a-1f7e92b6a852\") " pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:25:42.987223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/13d2df84-0465-42b2-9ff8-cb39aaa95e33-dbus\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:42.987223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986919    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/fe6ad5b4-7a10-49b2-9618-440018cb51c3-system-cni-dir\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.987223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986938    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a4b01391-be0e-4c48-a84d-2842bedb9248-sys\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.987223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.986973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b64a4088-eec0-4e59-bb26-7b4297a68fdf-host-slash\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.987223 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.987020    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/fe6ad5b4-7a10-49b2-9618-440018cb51c3-multus-daemon-config\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.987688 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.987667    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a4b01391-be0e-4c48-a84d-2842bedb9248-etc-tuned\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.994483 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.994438    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qksvq\" (UniqueName: \"kubernetes.io/projected/27821ce6-26d8-4a08-b587-91eeac050017-kube-api-access-qksvq\") pod \"node-resolver-5njds\" (UID: \"27821ce6-26d8-4a08-b587-91eeac050017\") " pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:42.994836 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.994812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vk9c\" (UniqueName: \"kubernetes.io/projected/a4b01391-be0e-4c48-a84d-2842bedb9248-kube-api-access-9vk9c\") pod \"tuned-tl99f\" (UID: \"a4b01391-be0e-4c48-a84d-2842bedb9248\") " pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:42.995078 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.995034    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72g6q\" (UniqueName: \"kubernetes.io/projected/fe6ad5b4-7a10-49b2-9618-440018cb51c3-kube-api-access-72g6q\") pod \"multus-lgh5k\" (UID: \"fe6ad5b4-7a10-49b2-9618-440018cb51c3\") " pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:42.995471 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.995445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7grn7\" (UniqueName: \"kubernetes.io/projected/b64a4088-eec0-4e59-bb26-7b4297a68fdf-kube-api-access-7grn7\") pod \"iptables-alerter-5wmkn\" (UID: \"b64a4088-eec0-4e59-bb26-7b4297a68fdf\") " pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:42.996088 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.996071    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c74gc\" (UniqueName: \"kubernetes.io/projected/8cfe68ed-2ecc-4369-adcd-235f6b199b24-kube-api-access-c74gc\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:42.996370 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:42.996351    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5r99\" (UniqueName: \"kubernetes.io/projected/ba892592-38ce-46ab-8073-907a0e715c39-kube-api-access-j5r99\") pod \"node-ca-p7867\" (UID: \"ba892592-38ce-46ab-8073-907a0e715c39\") " pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:43.070546 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.070506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c"
Jul 21 17:25:43.083353 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.083327    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:25:43.091193 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.091172    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-5xzbr"
Jul 21 17:25:43.100733 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.100711    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-tl99f"
Jul 21 17:25:43.108319 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.108302    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-p7867"
Jul 21 17:25:43.115620 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.115598    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-5njds"
Jul 21 17:25:43.122933 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.122914    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-5wmkn"
Jul 21 17:25:43.129596 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.129575    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-lgh5k"
Jul 21 17:25:43.134704 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.134687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:25:43.387471 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.387441    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:43.387643 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:43.387624    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:43.387702 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:43.387645    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:43.387702 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:43.387658    2563 projected.go:196] Error preparing data for projected volume kube-api-access-7qzhf for pod openshift-network-diagnostics/network-check-target-qq9ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:43.387818 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:43.387727    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf podName:ff525c04-45a3-46cd-883e-fbbc3416f73c nodeName:}" failed. No retries permitted until 2026-07-21 17:25:44.387706017 +0000 UTC m=+3.945713119 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qzhf" (UniqueName: "kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf") pod "network-check-target-qq9ln" (UID: "ff525c04-45a3-46cd-883e-fbbc3416f73c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:43.488175 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.488128    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:43.488354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.488226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:43.488354 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:43.488311    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:43.488354 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:43.488323    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:43.488488 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:43.488376    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs podName:8cfe68ed-2ecc-4369-adcd-235f6b199b24 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:44.488361774 +0000 UTC m=+4.046368863 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs") pod "network-metrics-daemon-rz8hm" (UID: "8cfe68ed-2ecc-4369-adcd-235f6b199b24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:43.488488 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:43.488393    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret podName:13d2df84-0465-42b2-9ff8-cb39aaa95e33 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:44.488384808 +0000 UTC m=+4.046391895 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret") pod "global-pull-secret-syncer-p8lvn" (UID: "13d2df84-0465-42b2-9ff8-cb39aaa95e33") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:43.756464 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:43.756437    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe6ad5b4_7a10_49b2_9618_440018cb51c3.slice/crio-e71b0eb189a1577f29223d6ce1060cb11b96d81ddea1db34ddec00fe10d95e09 WatchSource:0}: Error finding container e71b0eb189a1577f29223d6ce1060cb11b96d81ddea1db34ddec00fe10d95e09: Status 404 returned error can't find the container with id e71b0eb189a1577f29223d6ce1060cb11b96d81ddea1db34ddec00fe10d95e09
Jul 21 17:25:43.757653 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:43.757627    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4b01391_be0e_4c48_a84d_2842bedb9248.slice/crio-aa7b6d6284ccb172c9367952472b99c313a253732932b36760a4f75aac7dac0f WatchSource:0}: Error finding container aa7b6d6284ccb172c9367952472b99c313a253732932b36760a4f75aac7dac0f: Status 404 returned error can't find the container with id aa7b6d6284ccb172c9367952472b99c313a253732932b36760a4f75aac7dac0f
Jul 21 17:25:43.759887 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:43.759861    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb64a4088_eec0_4e59_bb26_7b4297a68fdf.slice/crio-da9571ecb26a5bafeec363de68fd4cd3a3660cdebb45d7d640cd00992b7071d4 WatchSource:0}: Error finding container da9571ecb26a5bafeec363de68fd4cd3a3660cdebb45d7d640cd00992b7071d4: Status 404 returned error can't find the container with id da9571ecb26a5bafeec363de68fd4cd3a3660cdebb45d7d640cd00992b7071d4
Jul 21 17:25:43.761679 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:43.760902    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7573a048_9229_4379_812a_1f7e92b6a852.slice/crio-35b7f29cb170020b86feb52d3a355a05540f530f4a6a0b1764199564ca08e39b WatchSource:0}: Error finding container 35b7f29cb170020b86feb52d3a355a05540f530f4a6a0b1764199564ca08e39b: Status 404 returned error can't find the container with id 35b7f29cb170020b86feb52d3a355a05540f530f4a6a0b1764199564ca08e39b
Jul 21 17:25:43.764212 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:25:43.764130    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9af59be9_5692_410e_a7b7_e15daa138583.slice/crio-40b215ffdc929b0985637cc06b64ec3c302e948b65df175fd34699e331004da3 WatchSource:0}: Error finding container 40b215ffdc929b0985637cc06b64ec3c302e948b65df175fd34699e331004da3: Status 404 returned error can't find the container with id 40b215ffdc929b0985637cc06b64ec3c302e948b65df175fd34699e331004da3
Jul 21 17:25:43.807776 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.807741    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 17:20:42 +0000 UTC" deadline="2027-12-23 09:05:25.328219575 +0000 UTC"
Jul 21 17:25:43.807884 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.807777    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12471h39m41.520446997s"
Jul 21 17:25:43.912810 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.912603    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal" event={"ID":"c4726a2cf6e4377be6ec06d1c802bab6","Type":"ContainerStarted","Data":"08f55831c72d25adce8f0be3828d4793243785369a7029d0a8edf67f7517ccc2"}
Jul 21 17:25:43.914037 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.914008    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"5c7b5ca66fcf0fc9af80f6aee3fc9233bbc124b18db7c9d1c5c38ed6fdeeb40a"}
Jul 21 17:25:43.915034 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.915005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" event={"ID":"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4","Type":"ContainerStarted","Data":"45a9294a5297bb41d0824f467370816356a8c775f42cc33a9222cec5b7a53b69"}
Jul 21 17:25:43.916161 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.916137    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v89wr" event={"ID":"7573a048-9229-4379-812a-1f7e92b6a852","Type":"ContainerStarted","Data":"35b7f29cb170020b86feb52d3a355a05540f530f4a6a0b1764199564ca08e39b"}
Jul 21 17:25:43.917444 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.917423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tl99f" event={"ID":"a4b01391-be0e-4c48-a84d-2842bedb9248","Type":"ContainerStarted","Data":"aa7b6d6284ccb172c9367952472b99c313a253732932b36760a4f75aac7dac0f"}
Jul 21 17:25:43.918636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.918615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p7867" event={"ID":"ba892592-38ce-46ab-8073-907a0e715c39","Type":"ContainerStarted","Data":"ab7208c8c62c1a0e3495e4ec615722d4fd640204f3b38643b09f11b08d9f495b"}
Jul 21 17:25:43.919616 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.919593    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5njds" event={"ID":"27821ce6-26d8-4a08-b587-91eeac050017","Type":"ContainerStarted","Data":"e7c94d5ddaa1cd642943aa24c117df386f28745c24dfbe2cbaaa83121dc6ed75"}
Jul 21 17:25:43.920667 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.920648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c" event={"ID":"9af59be9-5692-410e-a7b7-e15daa138583","Type":"ContainerStarted","Data":"40b215ffdc929b0985637cc06b64ec3c302e948b65df175fd34699e331004da3"}
Jul 21 17:25:43.921633 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.921613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5wmkn" event={"ID":"b64a4088-eec0-4e59-bb26-7b4297a68fdf","Type":"ContainerStarted","Data":"da9571ecb26a5bafeec363de68fd4cd3a3660cdebb45d7d640cd00992b7071d4"}
Jul 21 17:25:43.922612 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.922594    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lgh5k" event={"ID":"fe6ad5b4-7a10-49b2-9618-440018cb51c3","Type":"ContainerStarted","Data":"e71b0eb189a1577f29223d6ce1060cb11b96d81ddea1db34ddec00fe10d95e09"}
Jul 21 17:25:43.929426 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:43.929379    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-22.ec2.internal" podStartSLOduration=1.9293671 podStartE2EDuration="1.9293671s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:25:43.929131331 +0000 UTC m=+3.487138439" watchObservedRunningTime="2026-07-21 17:25:43.9293671 +0000 UTC m=+3.487374230"
Jul 21 17:25:44.393722 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.393679    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:44.393925 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.393867    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:44.393925 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.393891    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:44.393925 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.393903    2563 projected.go:196] Error preparing data for projected volume kube-api-access-7qzhf for pod openshift-network-diagnostics/network-check-target-qq9ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:44.394094 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.393963    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf podName:ff525c04-45a3-46cd-883e-fbbc3416f73c nodeName:}" failed. No retries permitted until 2026-07-21 17:25:46.393946102 +0000 UTC m=+5.951953205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qzhf" (UniqueName: "kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf") pod "network-check-target-qq9ln" (UID: "ff525c04-45a3-46cd-883e-fbbc3416f73c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:44.494273 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.494087    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:44.494273 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.494146    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:44.494273 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.494243    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:44.494564 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.494309    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs podName:8cfe68ed-2ecc-4369-adcd-235f6b199b24 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:46.494294237 +0000 UTC m=+6.052301322 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs") pod "network-metrics-daemon-rz8hm" (UID: "8cfe68ed-2ecc-4369-adcd-235f6b199b24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:44.494629 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.494250    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:44.494629 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.494615    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret podName:13d2df84-0465-42b2-9ff8-cb39aaa95e33 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:46.494600171 +0000 UTC m=+6.052607258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret") pod "global-pull-secret-syncer-p8lvn" (UID: "13d2df84-0465-42b2-9ff8-cb39aaa95e33") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:44.808345 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.808276    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-20 17:20:42 +0000 UTC" deadline="2027-12-17 16:53:10.278877166 +0000 UTC"
Jul 21 17:25:44.808345 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.808343    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12335h27m25.470539892s"
Jul 21 17:25:44.903260 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.903165    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:44.903503 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.903309    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:44.903856 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.903599    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:44.903856 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.903687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:44.903856 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.903807    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:25:44.904081 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:44.903903    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:44.963384 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.963347    2563 generic.go:356] "Generic (PLEG): container finished" podID="10fe471e90fa9c7c931061a10f11dce9" containerID="a2410327aa84ab1acb6e04e53ff187c6f43b96599fa1dd304a6801bc52c5d77a" exitCode=0
Jul 21 17:25:44.963542 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:44.963480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal" event={"ID":"10fe471e90fa9c7c931061a10f11dce9","Type":"ContainerDied","Data":"a2410327aa84ab1acb6e04e53ff187c6f43b96599fa1dd304a6801bc52c5d77a"}
Jul 21 17:25:45.987990 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:45.987952    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal" event={"ID":"10fe471e90fa9c7c931061a10f11dce9","Type":"ContainerStarted","Data":"b5a712151ebd804d01a33d62881e08bf34c2117bcd685996d2592d3f0592ddae"}
Jul 21 17:25:46.410660 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:46.410601    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:46.410875 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.410857    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:46.410940 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.410887    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:46.410940 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.410900    2563 projected.go:196] Error preparing data for projected volume kube-api-access-7qzhf for pod openshift-network-diagnostics/network-check-target-qq9ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:46.411042 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.410961    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf podName:ff525c04-45a3-46cd-883e-fbbc3416f73c nodeName:}" failed. No retries permitted until 2026-07-21 17:25:50.410942565 +0000 UTC m=+9.968949655 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qzhf" (UniqueName: "kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf") pod "network-check-target-qq9ln" (UID: "ff525c04-45a3-46cd-883e-fbbc3416f73c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:46.511148 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:46.511106    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:46.511331 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:46.511218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:46.511434 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.511392    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:46.511493 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.511465    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs podName:8cfe68ed-2ecc-4369-adcd-235f6b199b24 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:50.511443214 +0000 UTC m=+10.069450323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs") pod "network-metrics-daemon-rz8hm" (UID: "8cfe68ed-2ecc-4369-adcd-235f6b199b24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:46.511911 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.511890    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:46.512078 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.511948    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret podName:13d2df84-0465-42b2-9ff8-cb39aaa95e33 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:50.511932864 +0000 UTC m=+10.069939956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret") pod "global-pull-secret-syncer-p8lvn" (UID: "13d2df84-0465-42b2-9ff8-cb39aaa95e33") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:46.904405 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:46.903889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:46.904405 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:46.903895    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:46.904405 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.904023    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:25:46.904405 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:46.904050    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:46.904405 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.904132    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:46.904405 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:46.904219    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:48.903970 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:48.903901    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:48.904417 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:48.904027    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:48.904417 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:48.904324    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:48.904417 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:48.904394    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:48.904594 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:48.904499    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:48.904594 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:48.904587    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:25:50.444839 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:50.444784    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:50.445321 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.445002    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:50.445321 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.445023    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:50.445321 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.445035    2563 projected.go:196] Error preparing data for projected volume kube-api-access-7qzhf for pod openshift-network-diagnostics/network-check-target-qq9ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:50.445321 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.445100    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf podName:ff525c04-45a3-46cd-883e-fbbc3416f73c nodeName:}" failed. No retries permitted until 2026-07-21 17:25:58.445078769 +0000 UTC m=+18.003085861 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qzhf" (UniqueName: "kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf") pod "network-check-target-qq9ln" (UID: "ff525c04-45a3-46cd-883e-fbbc3416f73c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:50.546036 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:50.545984    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:50.546219 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:50.546050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:50.546219 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.546189    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:50.546325 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.546255    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret podName:13d2df84-0465-42b2-9ff8-cb39aaa95e33 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:58.546232149 +0000 UTC m=+18.104239242 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret") pod "global-pull-secret-syncer-p8lvn" (UID: "13d2df84-0465-42b2-9ff8-cb39aaa95e33") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:50.546559 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.546538    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:50.546630 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.546596    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs podName:8cfe68ed-2ecc-4369-adcd-235f6b199b24 nodeName:}" failed. No retries permitted until 2026-07-21 17:25:58.546579823 +0000 UTC m=+18.104586925 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs") pod "network-metrics-daemon-rz8hm" (UID: "8cfe68ed-2ecc-4369-adcd-235f6b199b24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:50.904075 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:50.904043    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:50.904267 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.904152    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:25:50.904536 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:50.904509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:50.904674 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.904609    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:50.904920 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:50.904758    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:50.904920 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:50.904886    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:52.903515 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:52.903478    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:52.903958 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:52.903646    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:52.903958 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:52.903891    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:52.904072 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:52.904009    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:25:52.904130 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:52.904077    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:52.904174 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:52.904153    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:54.902777 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:54.902739    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:54.902777 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:54.902773    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:54.903303 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:54.902886    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:25:54.903303 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:54.902894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:54.903303 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:54.902980    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:54.903303 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:54.903081    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:56.903496 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:56.903464    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:56.903978 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:56.903465    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:56.903978 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:56.903593    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:56.903978 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:56.903597    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:25:56.903978 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:56.903674    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:56.903978 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:56.903761    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:58.506847 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:58.506811    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:58.507384 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.506966    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:25:58.507384 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.506987    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:25:58.507384 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.507000    2563 projected.go:196] Error preparing data for projected volume kube-api-access-7qzhf for pod openshift-network-diagnostics/network-check-target-qq9ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:58.507384 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.507073    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf podName:ff525c04-45a3-46cd-883e-fbbc3416f73c nodeName:}" failed. No retries permitted until 2026-07-21 17:26:14.507045153 +0000 UTC m=+34.065052243 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qzhf" (UniqueName: "kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf") pod "network-check-target-qq9ln" (UID: "ff525c04-45a3-46cd-883e-fbbc3416f73c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:25:58.607414 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:58.607326    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:58.607414 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:58.607390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:58.607591 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.607498    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:58.607591 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.607526    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:58.607591 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.607562    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs podName:8cfe68ed-2ecc-4369-adcd-235f6b199b24 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:14.607546938 +0000 UTC m=+34.165554023 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs") pod "network-metrics-daemon-rz8hm" (UID: "8cfe68ed-2ecc-4369-adcd-235f6b199b24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:25:58.607591 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.607577    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret podName:13d2df84-0465-42b2-9ff8-cb39aaa95e33 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:14.6075711 +0000 UTC m=+34.165578185 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret") pod "global-pull-secret-syncer-p8lvn" (UID: "13d2df84-0465-42b2-9ff8-cb39aaa95e33") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:25:58.903584 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:58.903506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:25:58.903584 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:58.903517    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:25:58.903803 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:25:58.903506    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:25:58.903803 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.903638    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:25:58.903803 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.903770    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:25:58.903942 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:25:58.903913    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:00.904502 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:00.904463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:00.905004 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:00.904551    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:00.905004 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:00.904597    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:00.905004 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:00.904605    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:00.905004 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:00.904682    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:26:00.905004 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:00.904759    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:26:02.903737 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:02.903463    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:02.903737 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:02.903694    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:02.904324 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:02.903748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:02.904324 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:02.903832    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:26:02.904324 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:02.903889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:02.904324 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:02.904005    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:26:03.020950 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.020916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-p7867" event={"ID":"ba892592-38ce-46ab-8073-907a0e715c39","Type":"ContainerStarted","Data":"3df91919f449b6d89aed99a0e547fdb8ba760564a7e335a9618906a338c0e28a"}
Jul 21 17:26:03.022413 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.022387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-5njds" event={"ID":"27821ce6-26d8-4a08-b587-91eeac050017","Type":"ContainerStarted","Data":"722c1fbd11e45408509b08fd3d907c8442cb5681d7e99198358dd91926a1c356"}
Jul 21 17:26:03.023975 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.023947    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c" event={"ID":"9af59be9-5692-410e-a7b7-e15daa138583","Type":"ContainerStarted","Data":"df6717fcea856d0d5c6be6a7744350d9fdfc55138aab9341c8ca078ec2bccc35"}
Jul 21 17:26:03.025484 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.025460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-lgh5k" event={"ID":"fe6ad5b4-7a10-49b2-9618-440018cb51c3","Type":"ContainerStarted","Data":"0061e11411d3567634048384c35fcf7f78db4fa2644a03d09117e75f3acf91cf"}
Jul 21 17:26:03.027266 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.027248    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"4474e824a61d41d5618aff015b28b930612a96c5f3449dd6b5746f308f9ab202"}
Jul 21 17:26:03.027372 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.027300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"717cfb1fa10ba0e63b2cc6819365201805776c1d1e84553ecfad04242b32cdc0"}
Jul 21 17:26:03.028666 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.028647    2563 generic.go:356] "Generic (PLEG): container finished" podID="77eb9156-b7f3-4f07-88f2-c5a2329e7bc4" containerID="da20c159be10eb46667b5179d7650f9395b2cd01c5b38c09fbc4b3c857e8e9aa" exitCode=0
Jul 21 17:26:03.028751 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.028706    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" event={"ID":"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4","Type":"ContainerDied","Data":"da20c159be10eb46667b5179d7650f9395b2cd01c5b38c09fbc4b3c857e8e9aa"}
Jul 21 17:26:03.030337 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.030168    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-v89wr" event={"ID":"7573a048-9229-4379-812a-1f7e92b6a852","Type":"ContainerStarted","Data":"fe0c0cec4fa4a1582850bf1545a960bbf51038dc8134fdc168b4d106a1a17130"}
Jul 21 17:26:03.032030 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.032002    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-tl99f" event={"ID":"a4b01391-be0e-4c48-a84d-2842bedb9248","Type":"ContainerStarted","Data":"df8862416b740306a994d342d1a0e0a02eb44aa62937d10729e6625f0b649780"}
Jul 21 17:26:03.038384 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.038347    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-22.ec2.internal" podStartSLOduration=21.038336148 podStartE2EDuration="21.038336148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:25:46.006310371 +0000 UTC m=+5.564317479" watchObservedRunningTime="2026-07-21 17:26:03.038336148 +0000 UTC m=+22.596343255"
Jul 21 17:26:03.056550 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.056509    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-p7867" podStartSLOduration=3.177816013 podStartE2EDuration="22.056495823s" totalImagesPullingTime="18.87867981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:03.038024344 +0000 UTC m=+22.596031452" watchObservedRunningTime="2026-07-21 17:26:03.056495823 +0000 UTC m=+22.614502986"
Jul 21 17:26:03.056829 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.056777    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-tl99f" podStartSLOduration=4.141322712 podStartE2EDuration="23.056774142s" totalImagesPullingTime="18.91545143s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:03.056267647 +0000 UTC m=+22.614274753" watchObservedRunningTime="2026-07-21 17:26:03.056774142 +0000 UTC m=+22.614781246"
Jul 21 17:26:03.072295 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.072253    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-5njds" podStartSLOduration=3.193475279 podStartE2EDuration="22.072238132s" totalImagesPullingTime="18.878762853s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:03.072177966 +0000 UTC m=+22.630185072" watchObservedRunningTime="2026-07-21 17:26:03.072238132 +0000 UTC m=+22.630245234"
Jul 21 17:26:03.119429 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.119374    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-lgh5k" podStartSLOduration=3.177080049 podStartE2EDuration="22.119354356s" totalImagesPullingTime="18.942274307s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:03.11932312 +0000 UTC m=+22.677330227" watchObservedRunningTime="2026-07-21 17:26:03.119354356 +0000 UTC m=+22.677361464"
Jul 21 17:26:03.136047 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:03.136004    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-v89wr" podStartSLOduration=7.493966201 podStartE2EDuration="22.135986115s" totalImagesPullingTime="14.642019914s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:03.135146334 +0000 UTC m=+22.693153460" watchObservedRunningTime="2026-07-21 17:26:03.135986115 +0000 UTC m=+22.693993251"
Jul 21 17:26:04.036756 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.036515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-5wmkn" event={"ID":"b64a4088-eec0-4e59-bb26-7b4297a68fdf","Type":"ContainerStarted","Data":"5502be8b31ea19ea41352f6f076b2f8e81ce288ce20a9326915d717ff0af736e"}
Jul 21 17:26:04.040079 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.040057    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:26:04.040451 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.040421    2563 generic.go:356] "Generic (PLEG): container finished" podID="3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e" containerID="4474e824a61d41d5618aff015b28b930612a96c5f3449dd6b5746f308f9ab202" exitCode=1
Jul 21 17:26:04.040580 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.040511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerDied","Data":"4474e824a61d41d5618aff015b28b930612a96c5f3449dd6b5746f308f9ab202"}
Jul 21 17:26:04.040649 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.040600    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"ec21f791372a5ed09e5da780ce67e7b1079dbf415936a616088f10dd34fb7b6a"}
Jul 21 17:26:04.040704 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.040653    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"4bee7a42b3b2dcebc5c5ad0fa105a274641990176b1549dcbe59ceacf1384d4f"}
Jul 21 17:26:04.040758 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.040703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"fd22dc1656b5d086ea0cdb423a92a1e488a58f9486c9dfe6b9ec350686f20a18"}
Jul 21 17:26:04.040758 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.040751    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"6cf2a72ad270f0a392d52ccb322d03ff8ca7eb49e861782a9aa2a1744e835d27"}
Jul 21 17:26:04.055687 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.055642    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-5wmkn" podStartSLOduration=4.168596776 podStartE2EDuration="23.055620906s" totalImagesPullingTime="18.88702413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:04.054578976 +0000 UTC m=+23.612586083" watchObservedRunningTime="2026-07-21 17:26:04.055620906 +0000 UTC m=+23.613628015"
Jul 21 17:26:04.382106 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.382085    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 21 17:26:04.832967 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.832781    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-21T17:26:04.382102291Z","UUID":"86fc496b-09e4-433f-9eb3-72d9772d566d","Handler":null,"Name":"","Endpoint":""}
Jul 21 17:26:04.834543 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.834517    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 21 17:26:04.834693 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.834560    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 21 17:26:04.903723 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.903689    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:04.903902 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:04.903868    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:04.904248 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.904220    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:04.904334 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:04.904315    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:26:04.904399 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:04.904388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:04.904484 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:04.904467    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:26:05.045435 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:05.044954    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c" event={"ID":"9af59be9-5692-410e-a7b7-e15daa138583","Type":"ContainerStarted","Data":"27dd243915b97f38e0452059eb575821dd870b399915349c742bd653e449256d"}
Jul 21 17:26:06.049769 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:06.049728    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c" event={"ID":"9af59be9-5692-410e-a7b7-e15daa138583","Type":"ContainerStarted","Data":"2635733a903d9b1bac388cc66db0b8be75f89fafc65b68fdb907a96d60d5821b"}
Jul 21 17:26:06.053376 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:06.053354    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:26:06.053795 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:06.053767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"a3c5132dd111fc182eec95a53acd84efa29dcfd4e73bf10d638c51f74b5c8fec"}
Jul 21 17:26:06.069481 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:06.069427    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-clh7c" podStartSLOduration=4.689340275 podStartE2EDuration="26.069405024s" totalImagesPullingTime="21.380064749s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:40 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:06.069162982 +0000 UTC m=+25.627170108" watchObservedRunningTime="2026-07-21 17:26:06.069405024 +0000 UTC m=+25.627412131"
Jul 21 17:26:06.904821 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:06.903651    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:06.904821 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:06.903692    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:06.904821 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:06.903687    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:06.904821 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:06.903826    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:26:06.904821 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:06.904070    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:06.904821 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:06.904147    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:26:07.936722 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:07.936527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:26:07.937365 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:07.937249    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:26:08.061762 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.061735    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:26:08.062178 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.062154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"46d56e74c85694252d034bf774bff9baa19651a50c1a4e7a7bc535e7c28add6b"}
Jul 21 17:26:08.062431 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.062417    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:26:08.062511 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.062470    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:26:08.062610 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.062594    2563 scope.go:117] "RemoveContainer" containerID="4474e824a61d41d5618aff015b28b930612a96c5f3449dd6b5746f308f9ab202"
Jul 21 17:26:08.064237 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.064209    2563 generic.go:356] "Generic (PLEG): container finished" podID="77eb9156-b7f3-4f07-88f2-c5a2329e7bc4" containerID="d2ed5840a68acdadfc29ca3ffce9ed6aa44e520ca99eaf5f7cf3583efbb728f0" exitCode=0
Jul 21 17:26:08.064345 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.064290    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" event={"ID":"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4","Type":"ContainerDied","Data":"d2ed5840a68acdadfc29ca3ffce9ed6aa44e520ca99eaf5f7cf3583efbb728f0"}
Jul 21 17:26:08.064461 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.064418    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:26:08.064951 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.064934    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-v89wr"
Jul 21 17:26:08.079250 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.079240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:26:08.903628 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.903417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:08.903853 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:08.903726    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:26:08.903853 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.903434    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:08.903853 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:08.903849    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:26:08.904042 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:08.903417    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:08.904042 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:08.903930    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:09.070952 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:09.070929    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:26:09.071331 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:09.071305    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" event={"ID":"3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e","Type":"ContainerStarted","Data":"c8be73d9d2f8534776ccf9432432b7c315c9ef389892e5423dd5a98d6c938ca4"}
Jul 21 17:26:09.071667 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:09.071648    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:26:09.099262 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:09.099166    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:26:09.110353 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:09.110312    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf" podStartSLOduration=10.128306913 podStartE2EDuration="29.110299515s" totalImagesPullingTime="18.981992602s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:09.110047791 +0000 UTC m=+28.668054899" watchObservedRunningTime="2026-07-21 17:26:09.110299515 +0000 UTC m=+28.668306622"
Jul 21 17:26:10.075907 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:10.075871    2563 generic.go:356] "Generic (PLEG): container finished" podID="77eb9156-b7f3-4f07-88f2-c5a2329e7bc4" containerID="4476942664ade77261e32985474eff4d9b29a4095f8a96699cfbb30f74a0d6c3" exitCode=0
Jul 21 17:26:10.076437 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:10.075961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" event={"ID":"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4","Type":"ContainerDied","Data":"4476942664ade77261e32985474eff4d9b29a4095f8a96699cfbb30f74a0d6c3"}
Jul 21 17:26:10.109226 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:10.109204    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p8lvn"]
Jul 21 17:26:10.109352 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:10.109300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:10.109411 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:10.109384    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:10.111647 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:10.111441    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qq9ln"]
Jul 21 17:26:10.111647 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:10.111533    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:10.111647 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:10.111612    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:26:10.112645 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:10.112625    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rz8hm"]
Jul 21 17:26:10.112727 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:10.112703    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:10.112783 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:10.112769    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:26:11.903664 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:11.903635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:11.904102 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:11.903674    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:11.904102 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:11.903638    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:11.904102 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:11.903776    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:26:11.904102 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:11.903869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:11.904102 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:11.903920    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:26:12.082717 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:12.082687    2563 generic.go:356] "Generic (PLEG): container finished" podID="77eb9156-b7f3-4f07-88f2-c5a2329e7bc4" containerID="a69fdbad5e8cac57e48e102008ce98b23252d1a54c958c10bbdec0c21cd8ad62" exitCode=0
Jul 21 17:26:12.082855 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:12.082720    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" event={"ID":"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4","Type":"ContainerDied","Data":"a69fdbad5e8cac57e48e102008ce98b23252d1a54c958c10bbdec0c21cd8ad62"}
Jul 21 17:26:13.903537 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:13.903286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:13.903944 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:13.903307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:13.903944 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:13.903638    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-p8lvn" podUID="13d2df84-0465-42b2-9ff8-cb39aaa95e33"
Jul 21 17:26:13.903944 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:13.903716    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-qq9ln" podUID="ff525c04-45a3-46cd-883e-fbbc3416f73c"
Jul 21 17:26:13.903944 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:13.903314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:13.903944 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:13.903811    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-rz8hm" podUID="8cfe68ed-2ecc-4369-adcd-235f6b199b24"
Jul 21 17:26:14.538570 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:14.538534    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:14.538769 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:14.538683    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 21 17:26:14.538769 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:14.538698    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 21 17:26:14.538769 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:14.538710    2563 projected.go:196] Error preparing data for projected volume kube-api-access-7qzhf for pod openshift-network-diagnostics/network-check-target-qq9ln: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:14.538769 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:14.538763    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf podName:ff525c04-45a3-46cd-883e-fbbc3416f73c nodeName:}" failed. No retries permitted until 2026-07-21 17:26:46.53875085 +0000 UTC m=+66.096757936 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-7qzhf" (UniqueName: "kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf") pod "network-check-target-qq9ln" (UID: "ff525c04-45a3-46cd-883e-fbbc3416f73c") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 21 17:26:14.639702 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:14.639668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:14.639902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:14.639723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:14.639902 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:14.639829    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:14.639902 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:14.639900    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs podName:8cfe68ed-2ecc-4369-adcd-235f6b199b24 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:46.639879453 +0000 UTC m=+66.197886542 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs") pod "network-metrics-daemon-rz8hm" (UID: "8cfe68ed-2ecc-4369-adcd-235f6b199b24") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 21 17:26:14.640037 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:14.639905    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:14.640037 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:14.639937    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret podName:13d2df84-0465-42b2-9ff8-cb39aaa95e33 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:46.639927531 +0000 UTC m=+66.197934616 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret") pod "global-pull-secret-syncer-p8lvn" (UID: "13d2df84-0465-42b2-9ff8-cb39aaa95e33") : object "kube-system"/"original-pull-secret" not registered
Jul 21 17:26:15.829735 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.829691    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-22.ec2.internal" event="NodeReady"
Jul 21 17:26:15.830216 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.829866    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 21 17:26:15.903307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.903267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:15.903307 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.903301    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:15.903542 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.903303    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:15.909132 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.909112    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 17:26:15.909636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.909618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 17:26:15.909900 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.909885    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 17:26:15.913086 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.911495    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 17:26:15.913086 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.911748    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swxfv\""
Jul 21 17:26:15.913086 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.912103    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gndbp\""
Jul 21 17:26:15.913086 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.912431    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qmpzt"]
Jul 21 17:26:15.941811 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.941771    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-hqmpp"]
Jul 21 17:26:15.941992 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.941976    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:15.946236 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.946046    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c7lxh\""
Jul 21 17:26:15.946584 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.946563    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 21 17:26:15.946584 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.946579    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 21 17:26:15.959639 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.959622    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmpzt"]
Jul 21 17:26:15.959727 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.959654    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hqmpp"]
Jul 21 17:26:15.959769 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.959742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hqmpp"
Jul 21 17:26:15.965178 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.964851    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 21 17:26:15.965178 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.965040    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 21 17:26:15.965178 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.965080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-d5qz9\""
Jul 21 17:26:15.965178 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:15.965086    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 21 17:26:16.049440 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.049407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a92e5b-3446-457c-bda1-e069f7fedc95-config-volume\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.049605 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.049479    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a92e5b-3446-457c-bda1-e069f7fedc95-tmp-dir\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.049605 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.049539    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2bhm\" (UniqueName: \"kubernetes.io/projected/e1a92e5b-3446-457c-bda1-e069f7fedc95-kube-api-access-t2bhm\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.049605 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.049596    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.049759 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.049624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46866\" (UniqueName: \"kubernetes.io/projected/83f3e42b-06f3-4ffa-8270-a5b076a775fc-kube-api-access-46866\") pod \"ingress-canary-hqmpp\" (UID: \"83f3e42b-06f3-4ffa-8270-a5b076a775fc\") " pod="openshift-ingress-canary/ingress-canary-hqmpp"
Jul 21 17:26:16.049759 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.049653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83f3e42b-06f3-4ffa-8270-a5b076a775fc-cert\") pod \"ingress-canary-hqmpp\" (UID: \"83f3e42b-06f3-4ffa-8270-a5b076a775fc\") " pod="openshift-ingress-canary/ingress-canary-hqmpp"
Jul 21 17:26:16.150712 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.150626    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2bhm\" (UniqueName: \"kubernetes.io/projected/e1a92e5b-3446-457c-bda1-e069f7fedc95-kube-api-access-t2bhm\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.150712 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.150701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.150961 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.150731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46866\" (UniqueName: \"kubernetes.io/projected/83f3e42b-06f3-4ffa-8270-a5b076a775fc-kube-api-access-46866\") pod \"ingress-canary-hqmpp\" (UID: \"83f3e42b-06f3-4ffa-8270-a5b076a775fc\") " pod="openshift-ingress-canary/ingress-canary-hqmpp"
Jul 21 17:26:16.150961 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.150751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83f3e42b-06f3-4ffa-8270-a5b076a775fc-cert\") pod \"ingress-canary-hqmpp\" (UID: \"83f3e42b-06f3-4ffa-8270-a5b076a775fc\") " pod="openshift-ingress-canary/ingress-canary-hqmpp"
Jul 21 17:26:16.150961 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.150771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a92e5b-3446-457c-bda1-e069f7fedc95-config-volume\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.150961 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.150837    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a92e5b-3446-457c-bda1-e069f7fedc95-tmp-dir\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.150961 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:16.150911    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:16.151178 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:16.150986    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls podName:e1a92e5b-3446-457c-bda1-e069f7fedc95 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:16.650965218 +0000 UTC m=+36.208972306 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls") pod "dns-default-qmpzt" (UID: "e1a92e5b-3446-457c-bda1-e069f7fedc95") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:16.151227 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.151207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e1a92e5b-3446-457c-bda1-e069f7fedc95-tmp-dir\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.151442 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.151405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/e1a92e5b-3446-457c-bda1-e069f7fedc95-config-volume\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.155672 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.155634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/83f3e42b-06f3-4ffa-8270-a5b076a775fc-cert\") pod \"ingress-canary-hqmpp\" (UID: \"83f3e42b-06f3-4ffa-8270-a5b076a775fc\") " pod="openshift-ingress-canary/ingress-canary-hqmpp"
Jul 21 17:26:16.161020 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.160993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46866\" (UniqueName: \"kubernetes.io/projected/83f3e42b-06f3-4ffa-8270-a5b076a775fc-kube-api-access-46866\") pod \"ingress-canary-hqmpp\" (UID: \"83f3e42b-06f3-4ffa-8270-a5b076a775fc\") " pod="openshift-ingress-canary/ingress-canary-hqmpp"
Jul 21 17:26:16.161263 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.161245    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2bhm\" (UniqueName: \"kubernetes.io/projected/e1a92e5b-3446-457c-bda1-e069f7fedc95-kube-api-access-t2bhm\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.268179 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.268149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-hqmpp"
Jul 21 17:26:16.497998 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.497966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-hqmpp"]
Jul 21 17:26:16.568467 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:26:16.568440    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83f3e42b_06f3_4ffa_8270_a5b076a775fc.slice/crio-0ed13085ab5e5ebf3c8228bb5c736aa3c362fb39ea6f415493c8a5fdb88551ab WatchSource:0}: Error finding container 0ed13085ab5e5ebf3c8228bb5c736aa3c362fb39ea6f415493c8a5fdb88551ab: Status 404 returned error can't find the container with id 0ed13085ab5e5ebf3c8228bb5c736aa3c362fb39ea6f415493c8a5fdb88551ab
Jul 21 17:26:16.658025 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:16.655063    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:16.658025 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:16.655228    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:16.658025 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:16.655387    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls podName:e1a92e5b-3446-457c-bda1-e069f7fedc95 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:17.655359197 +0000 UTC m=+37.213366293 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls") pod "dns-default-qmpzt" (UID: "e1a92e5b-3446-457c-bda1-e069f7fedc95") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:17.095521 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:17.095480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hqmpp" event={"ID":"83f3e42b-06f3-4ffa-8270-a5b076a775fc","Type":"ContainerStarted","Data":"0ed13085ab5e5ebf3c8228bb5c736aa3c362fb39ea6f415493c8a5fdb88551ab"}
Jul 21 17:26:17.098605 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:17.098578    2563 generic.go:356] "Generic (PLEG): container finished" podID="77eb9156-b7f3-4f07-88f2-c5a2329e7bc4" containerID="2b44991c37e1eaa11a9b7b5b08dce39381dc34dcb3365400c4c02b0b0a682492" exitCode=0
Jul 21 17:26:17.098734 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:17.098625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" event={"ID":"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4","Type":"ContainerDied","Data":"2b44991c37e1eaa11a9b7b5b08dce39381dc34dcb3365400c4c02b0b0a682492"}
Jul 21 17:26:17.665624 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:17.665584    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:17.665852 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:17.665764    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:17.665919 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:17.665855    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls podName:e1a92e5b-3446-457c-bda1-e069f7fedc95 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:19.665833783 +0000 UTC m=+39.223840883 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls") pod "dns-default-qmpzt" (UID: "e1a92e5b-3446-457c-bda1-e069f7fedc95") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:18.105105 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:18.105066    2563 generic.go:356] "Generic (PLEG): container finished" podID="77eb9156-b7f3-4f07-88f2-c5a2329e7bc4" containerID="38cf0cd62ec6cac1b2b98cf6478ab77a45364aa1157b105066103bd7bca6bb0b" exitCode=0
Jul 21 17:26:18.105496 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:18.105112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" event={"ID":"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4","Type":"ContainerDied","Data":"38cf0cd62ec6cac1b2b98cf6478ab77a45364aa1157b105066103bd7bca6bb0b"}
Jul 21 17:26:19.108954 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:19.108915    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-hqmpp" event={"ID":"83f3e42b-06f3-4ffa-8270-a5b076a775fc","Type":"ContainerStarted","Data":"ead152bec042afed24b6d0fd78d1c826bcbd57682c2659d221fff26ee1cc9a2c"}
Jul 21 17:26:19.112593 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:19.112567    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" event={"ID":"77eb9156-b7f3-4f07-88f2-c5a2329e7bc4","Type":"ContainerStarted","Data":"6410867f91b9e8559946f3865d6854730eb7b47f7c93ac4e6354e4491f02c36e"}
Jul 21 17:26:19.131070 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:19.131031    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-hqmpp" podStartSLOduration=2.053261122 podStartE2EDuration="4.131010275s" totalImagesPullingTime="2.077749153s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:19.130802216 +0000 UTC m=+38.688809315" watchObservedRunningTime="2026-07-21 17:26:19.131010275 +0000 UTC m=+38.689017382"
Jul 21 17:26:19.166038 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:19.165995    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-5xzbr" podStartSLOduration=9.011162061 podStartE2EDuration="39.165977133s" totalImagesPullingTime="30.154815072s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:40 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:19.163526061 +0000 UTC m=+38.721533193" watchObservedRunningTime="2026-07-21 17:26:19.165977133 +0000 UTC m=+38.723984241"
Jul 21 17:26:19.680839 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:19.680651    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:19.681011 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:19.680849    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:19.681011 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:19.680933    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls podName:e1a92e5b-3446-457c-bda1-e069f7fedc95 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:23.680908753 +0000 UTC m=+43.238915841 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls") pod "dns-default-qmpzt" (UID: "e1a92e5b-3446-457c-bda1-e069f7fedc95") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:23.709776 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:23.709738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:23.710166 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:23.709893    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:23.710166 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:23.709950    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls podName:e1a92e5b-3446-457c-bda1-e069f7fedc95 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:31.709935779 +0000 UTC m=+51.267942863 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls") pod "dns-default-qmpzt" (UID: "e1a92e5b-3446-457c-bda1-e069f7fedc95") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:31.763406 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:31.763359    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:31.764035 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:31.763529    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:31.764035 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:31.763617    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls podName:e1a92e5b-3446-457c-bda1-e069f7fedc95 nodeName:}" failed. No retries permitted until 2026-07-21 17:26:47.763595237 +0000 UTC m=+67.321602321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls") pod "dns-default-qmpzt" (UID: "e1a92e5b-3446-457c-bda1-e069f7fedc95") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:41.088450 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:41.088421    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-shlvf"
Jul 21 17:26:46.558294 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.558247    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:46.561242 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.561221    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 21 17:26:46.572076 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.572056    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 21 17:26:46.583521 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.583497    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qzhf\" (UniqueName: \"kubernetes.io/projected/ff525c04-45a3-46cd-883e-fbbc3416f73c-kube-api-access-7qzhf\") pod \"network-check-target-qq9ln\" (UID: \"ff525c04-45a3-46cd-883e-fbbc3416f73c\") " pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:46.658635 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.658598    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:26:46.658836 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.658643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:46.661572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.661552    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 21 17:26:46.661630 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.661612    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 21 17:26:46.669830 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:46.669807    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 21 17:26:46.669907 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:46.669888    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs podName:8cfe68ed-2ecc-4369-adcd-235f6b199b24 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:50.669864559 +0000 UTC m=+130.227871645 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs") pod "network-metrics-daemon-rz8hm" (UID: "8cfe68ed-2ecc-4369-adcd-235f6b199b24") : secret "metrics-daemon-secret" not found
Jul 21 17:26:46.672536 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.672510    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/13d2df84-0465-42b2-9ff8-cb39aaa95e33-original-pull-secret\") pod \"global-pull-secret-syncer-p8lvn\" (UID: \"13d2df84-0465-42b2-9ff8-cb39aaa95e33\") " pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:46.817921 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.817782    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-p8lvn"
Jul 21 17:26:46.827552 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.827526    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-gndbp\""
Jul 21 17:26:46.834764 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.834741    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:46.969192 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.969162    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-p8lvn"]
Jul 21 17:26:46.973276 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:26:46.973254    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d2df84_0465_42b2_9ff8_cb39aaa95e33.slice/crio-206131d6db44bf2c9d9a843b3f16568b52d62ee4f13d8d9ddf60cb1d09145465 WatchSource:0}: Error finding container 206131d6db44bf2c9d9a843b3f16568b52d62ee4f13d8d9ddf60cb1d09145465: Status 404 returned error can't find the container with id 206131d6db44bf2c9d9a843b3f16568b52d62ee4f13d8d9ddf60cb1d09145465
Jul 21 17:26:46.982450 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:46.982418    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-qq9ln"]
Jul 21 17:26:46.985598 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:26:46.985571    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff525c04_45a3_46cd_883e_fbbc3416f73c.slice/crio-91b4cc13d90d868474075426a01db4e356f7086254e8f8a47787d332dfbca02d WatchSource:0}: Error finding container 91b4cc13d90d868474075426a01db4e356f7086254e8f8a47787d332dfbca02d: Status 404 returned error can't find the container with id 91b4cc13d90d868474075426a01db4e356f7086254e8f8a47787d332dfbca02d
Jul 21 17:26:47.181124 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:47.181027    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qq9ln" event={"ID":"ff525c04-45a3-46cd-883e-fbbc3416f73c","Type":"ContainerStarted","Data":"91b4cc13d90d868474075426a01db4e356f7086254e8f8a47787d332dfbca02d"}
Jul 21 17:26:47.182045 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:47.182020    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p8lvn" event={"ID":"13d2df84-0465-42b2-9ff8-cb39aaa95e33","Type":"ContainerStarted","Data":"206131d6db44bf2c9d9a843b3f16568b52d62ee4f13d8d9ddf60cb1d09145465"}
Jul 21 17:26:47.768619 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:47.768576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:26:47.769120 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:47.768728    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:26:47.769120 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:26:47.768815    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls podName:e1a92e5b-3446-457c-bda1-e069f7fedc95 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:19.768776767 +0000 UTC m=+99.326783864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls") pod "dns-default-qmpzt" (UID: "e1a92e5b-3446-457c-bda1-e069f7fedc95") : secret "dns-default-metrics-tls" not found
Jul 21 17:26:52.198060 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:52.197963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-qq9ln" event={"ID":"ff525c04-45a3-46cd-883e-fbbc3416f73c","Type":"ContainerStarted","Data":"0c78aeeda2b9665587d1147d197a132fb3129cc8d9134a4c1a0201a7954e4b6f"}
Jul 21 17:26:52.198512 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:52.198060    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:26:52.199421 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:52.199396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-p8lvn" event={"ID":"13d2df84-0465-42b2-9ff8-cb39aaa95e33","Type":"ContainerStarted","Data":"6a3d99560228c0f6c0300d25507b199e7386554a217c3c8b255149aabacb7a4f"}
Jul 21 17:26:52.217609 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:52.217565    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-qq9ln" podStartSLOduration=67.283294568 podStartE2EDuration="1m12.217550021s" totalImagesPullingTime="4.934255453s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:52.21752882 +0000 UTC m=+71.775535941" watchObservedRunningTime="2026-07-21 17:26:52.217550021 +0000 UTC m=+71.775557129"
Jul 21 17:26:52.236018 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:26:52.235978    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-p8lvn" podStartSLOduration=66.278228488 podStartE2EDuration="1m11.23596382s" totalImagesPullingTime="4.957735332s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:25:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:26:52.23528651 +0000 UTC m=+71.793293616" watchObservedRunningTime="2026-07-21 17:26:52.23596382 +0000 UTC m=+71.793970935"
Jul 21 17:27:02.500516 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.500400    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"]
Jul 21 17:27:02.502324 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.502304    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:02.503473 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.503451    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-7cfbdb577d-z6pf4"]
Jul 21 17:27:02.506296 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.505576    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.506296 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.506124    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 21 17:27:02.507750 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.506577    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 21 17:27:02.507750 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.507686    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 21 17:27:02.509232 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.509216    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-xwk5l\""
Jul 21 17:27:02.509232 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.509225    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 21 17:27:02.511340 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.511322    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 21 17:27:02.511546 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.511527    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-4qnf6\""
Jul 21 17:27:02.511655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.511565    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 21 17:27:02.511655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.511643    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 21 17:27:02.511825 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.511674    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 21 17:27:02.511894 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.511880    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 21 17:27:02.511937 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.511894    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 21 17:27:02.518490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.518470    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"]
Jul 21 17:27:02.525927 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.525907    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7cfbdb577d-z6pf4"]
Jul 21 17:27:02.566439 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.566407    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:02.566578 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.566449    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-stats-auth\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.566578 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.566474    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8e304a3-94a7-4671-a188-fbc216c91b6f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:02.566578 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.566529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzp5j\" (UniqueName: \"kubernetes.io/projected/15e82759-7714-4b28-b4cb-c9289ae76056-kube-api-access-rzp5j\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.566578 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.566563    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.566815 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.566599    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-default-certificate\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.566815 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.566626    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.566815 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.566666    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k4hgd\" (UniqueName: \"kubernetes.io/projected/b8e304a3-94a7-4671-a188-fbc216c91b6f-kube-api-access-k4hgd\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:02.667187 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.667154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-default-certificate\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.667187 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.667187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.667360 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.667217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k4hgd\" (UniqueName: \"kubernetes.io/projected/b8e304a3-94a7-4671-a188-fbc216c91b6f-kube-api-access-k4hgd\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:02.667360 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.667253    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:02.667360 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.667274    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-stats-auth\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.667360 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.667290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8e304a3-94a7-4671-a188-fbc216c91b6f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:02.667360 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:02.667297    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:27:02.667603 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:02.667367    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:03.167346829 +0000 UTC m=+82.725353914 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : secret "router-metrics-certs-default" not found
Jul 21 17:27:02.667603 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:02.667377    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:02.667603 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.667398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzp5j\" (UniqueName: \"kubernetes.io/projected/15e82759-7714-4b28-b4cb-c9289ae76056-kube-api-access-rzp5j\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.667603 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:02.667441    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls podName:b8e304a3-94a7-4671-a188-fbc216c91b6f nodeName:}" failed. No retries permitted until 2026-07-21 17:27:03.167425261 +0000 UTC m=+82.725432346 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-4fj4r" (UID: "b8e304a3-94a7-4671-a188-fbc216c91b6f") : secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:02.667603 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.667461    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.667603 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:02.667582    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:03.167570351 +0000 UTC m=+82.725577437 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : configmap references non-existent config key: service-ca.crt
Jul 21 17:27:02.668172 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.668154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/b8e304a3-94a7-4671-a188-fbc216c91b6f-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:02.669813 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.669777    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-stats-auth\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.669869 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.669854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-default-certificate\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.678586 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.678561    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzp5j\" (UniqueName: \"kubernetes.io/projected/15e82759-7714-4b28-b4cb-c9289ae76056-kube-api-access-rzp5j\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:02.678857 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:02.678837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k4hgd\" (UniqueName: \"kubernetes.io/projected/b8e304a3-94a7-4671-a188-fbc216c91b6f-kube-api-access-k4hgd\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:03.170887 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:03.170852    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:03.170887 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:03.170895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:03.171158 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:03.170930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:03.171158 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:03.171015    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:04.170996844 +0000 UTC m=+83.729003948 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : configmap references non-existent config key: service-ca.crt
Jul 21 17:27:03.171158 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:03.171024    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:03.171158 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:03.171042    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:27:03.171158 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:03.171062    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls podName:b8e304a3-94a7-4671-a188-fbc216c91b6f nodeName:}" failed. No retries permitted until 2026-07-21 17:27:04.171052087 +0000 UTC m=+83.729059176 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-4fj4r" (UID: "b8e304a3-94a7-4671-a188-fbc216c91b6f") : secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:03.171158 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:03.171095    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:04.171076194 +0000 UTC m=+83.729083279 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : secret "router-metrics-certs-default" not found
Jul 21 17:27:04.178179 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:04.178139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:04.178557 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:04.178203    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:04.178557 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:04.178248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:04.178557 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:04.178293    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:27:04.178557 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:04.178350    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:04.178557 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:04.178359    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:06.178345179 +0000 UTC m=+85.736352264 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : secret "router-metrics-certs-default" not found
Jul 21 17:27:04.178557 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:04.178396    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:06.17838508 +0000 UTC m=+85.736392165 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : configmap references non-existent config key: service-ca.crt
Jul 21 17:27:04.178557 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:04.178409    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls podName:b8e304a3-94a7-4671-a188-fbc216c91b6f nodeName:}" failed. No retries permitted until 2026-07-21 17:27:06.178401932 +0000 UTC m=+85.736409016 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-4fj4r" (UID: "b8e304a3-94a7-4671-a188-fbc216c91b6f") : secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:06.189959 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:06.189925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:06.189959 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:06.189966    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:06.190442 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:06.190080    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:27:06.190442 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:06.190094    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:10.190076336 +0000 UTC m=+89.748083425 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : configmap references non-existent config key: service-ca.crt
Jul 21 17:27:06.190442 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:06.190118    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:10.190107296 +0000 UTC m=+89.748114380 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : secret "router-metrics-certs-default" not found
Jul 21 17:27:06.190442 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:06.190131    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:06.190442 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:06.190184    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:06.190442 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:06.190212    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls podName:b8e304a3-94a7-4671-a188-fbc216c91b6f nodeName:}" failed. No retries permitted until 2026-07-21 17:27:10.190205551 +0000 UTC m=+89.748212635 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-4fj4r" (UID: "b8e304a3-94a7-4671-a188-fbc216c91b6f") : secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:07.661013 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.660980    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-dl229"]
Jul 21 17:27:07.663404 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.663387    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.667739 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.667718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 21 17:27:07.668732 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.668708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-pmcsw\""
Jul 21 17:27:07.669251 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.669234    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 21 17:27:07.669412 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.669393    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 21 17:27:07.669856 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.669841    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 21 17:27:07.686183 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.686161    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dl229"]
Jul 21 17:27:07.700486 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.700465    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1350fcd2-f330-46c1-a54d-06f482e0800a-data-volume\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.700584 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.700501    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1350fcd2-f330-46c1-a54d-06f482e0800a-crio-socket\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.700584 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.700521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.700665 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.700617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.700705 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.700682    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-csl79\" (UniqueName: \"kubernetes.io/projected/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-api-access-csl79\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.801528 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.801498    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1350fcd2-f330-46c1-a54d-06f482e0800a-data-volume\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.801655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.801540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1350fcd2-f330-46c1-a54d-06f482e0800a-crio-socket\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.801655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.801560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.801655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.801599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.801655 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.801634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/1350fcd2-f330-46c1-a54d-06f482e0800a-crio-socket\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.801784 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.801639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-csl79\" (UniqueName: \"kubernetes.io/projected/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-api-access-csl79\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.801784 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:07.801717    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:07.801784 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:07.801732    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:07.801784 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:07.801776    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:08.301756814 +0000 UTC m=+87.859763899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:07.801979 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:07.801811    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:08.301804481 +0000 UTC m=+87.859811566 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:07.801979 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.801824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/1350fcd2-f330-46c1-a54d-06f482e0800a-data-volume\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:07.817558 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:07.817528    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-csl79\" (UniqueName: \"kubernetes.io/projected/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-api-access-csl79\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:08.305545 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:08.305504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:08.305705 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:08.305577    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:08.305705 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:08.305649    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:08.305705 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:08.305664    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:08.305827 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:08.305728    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:09.30570693 +0000 UTC m=+88.863714026 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:08.305827 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:08.305746    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:09.305737173 +0000 UTC m=+88.863744258 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:09.312667 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:09.312634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:09.313167 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:09.312691    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:09.313167 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:09.312752    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:09.313167 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:09.312775    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:09.313167 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:09.312827    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:11.312811963 +0000 UTC m=+90.870819048 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:09.313167 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:09.312848    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:11.312834512 +0000 UTC m=+90.870841597 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:09.399842 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:09.399821    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5njds_27821ce6-26d8-4a08-b587-91eeac050017/dns-node-resolver/0.log"
Jul 21 17:27:10.218266 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:10.218231    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:10.218452 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:10.218272    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:10.218452 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:10.218315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:10.218452 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:10.218393    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:18.218376669 +0000 UTC m=+97.776383754 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : configmap references non-existent config key: service-ca.crt
Jul 21 17:27:10.218452 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:10.218431    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:10.218452 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:10.218447    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:27:10.218658 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:10.218474    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls podName:b8e304a3-94a7-4671-a188-fbc216c91b6f nodeName:}" failed. No retries permitted until 2026-07-21 17:27:18.218463262 +0000 UTC m=+97.776470347 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-4fj4r" (UID: "b8e304a3-94a7-4671-a188-fbc216c91b6f") : secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:10.218658 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:10.218498    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:18.218483669 +0000 UTC m=+97.776490757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : secret "router-metrics-certs-default" not found
Jul 21 17:27:10.398958 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:10.398932    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p7867_ba892592-38ce-46ab-8073-907a0e715c39/node-ca/0.log"
Jul 21 17:27:10.996720 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:10.996695    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hqmpp_83f3e42b-06f3-4ffa-8270-a5b076a775fc/serve-healthcheck-canary/0.log"
Jul 21 17:27:11.326768 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:11.326688    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:11.326768 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:11.326757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:11.326990 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:11.326833    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:11.326990 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:11.326862    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:11.326990 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:11.326884    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:15.326869923 +0000 UTC m=+94.884877008 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:11.326990 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:11.326911    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:15.326897911 +0000 UTC m=+94.884904996 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:15.357102 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:15.357073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:15.357641 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:15.357129    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:15.357641 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:15.357198    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 21 17:27:15.357641 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:15.357225    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:15.357641 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:15.357269    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:23.357250413 +0000 UTC m=+102.915257497 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:15.357641 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:15.357287    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:23.357279991 +0000 UTC m=+102.915287075 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : configmap "kube-rbac-proxy" not found
Jul 21 17:27:17.189615 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.189583    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z"]
Jul 21 17:27:17.191625 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.191605    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z"
Jul 21 17:27:17.194459 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.194442    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-vknkf\""
Jul 21 17:27:17.202321 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.202301    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z"]
Jul 21 17:27:17.270779 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.270748    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xpdqz\" (UniqueName: \"kubernetes.io/projected/87803527-5457-4f07-9fa4-54080f6959a2-kube-api-access-xpdqz\") pod \"network-check-source-7bc5b6bf85-l5q5z\" (UID: \"87803527-5457-4f07-9fa4-54080f6959a2\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z"
Jul 21 17:27:17.371677 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.371648    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xpdqz\" (UniqueName: \"kubernetes.io/projected/87803527-5457-4f07-9fa4-54080f6959a2-kube-api-access-xpdqz\") pod \"network-check-source-7bc5b6bf85-l5q5z\" (UID: \"87803527-5457-4f07-9fa4-54080f6959a2\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z"
Jul 21 17:27:17.380191 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.380168    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xpdqz\" (UniqueName: \"kubernetes.io/projected/87803527-5457-4f07-9fa4-54080f6959a2-kube-api-access-xpdqz\") pod \"network-check-source-7bc5b6bf85-l5q5z\" (UID: \"87803527-5457-4f07-9fa4-54080f6959a2\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z"
Jul 21 17:27:17.410934 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.410912    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5574d67986-vk42j"]
Jul 21 17:27:17.412722 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.412708    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.415357 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.415335    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 21 17:27:17.415482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.415464    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 21 17:27:17.415555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.415542    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 21 17:27:17.415646 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.415634    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-j5prf\""
Jul 21 17:27:17.420695 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.420677    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 21 17:27:17.429099 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.429077    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5574d67986-vk42j"]
Jul 21 17:27:17.473006 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.472940    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31c70f4-3128-4aa3-b468-29c378079b26-ca-trust-extracted\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.473103 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.473053    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-image-registry-private-configuration\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.473103 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.473081    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-trusted-ca\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.473182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.473114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-bound-sa-token\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.473182 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.473140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h228x\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-kube-api-access-h228x\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.473260 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.473178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-registry-certificates\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.473260 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.473252    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-installation-pull-secrets\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.473336 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.473271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.500649 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.500620    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z"
Jul 21 17:27:17.574328 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.574295    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-trusted-ca\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.574450 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.574349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-bound-sa-token\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.574450 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.574384    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h228x\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-kube-api-access-h228x\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.574659 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.574634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-registry-certificates\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.574745 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.574701    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-installation-pull-secrets\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.574815 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.574745    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.574919 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.574839    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31c70f4-3128-4aa3-b468-29c378079b26-ca-trust-extracted\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.574972 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.574954    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-image-registry-private-configuration\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.575669 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.575330    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-registry-certificates\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.575669 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:17.575355    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:27:17.575669 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:17.575374    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5574d67986-vk42j: secret "image-registry-tls" not found
Jul 21 17:27:17.575669 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:17.575443    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls podName:b31c70f4-3128-4aa3-b468-29c378079b26 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:18.075422988 +0000 UTC m=+97.633430100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls") pod "image-registry-5574d67986-vk42j" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26") : secret "image-registry-tls" not found
Jul 21 17:27:17.575669 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.575617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31c70f4-3128-4aa3-b468-29c378079b26-ca-trust-extracted\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.576027 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.575894    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-trusted-ca\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.577497 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.577476    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-image-registry-private-configuration\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.578122 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.578097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-installation-pull-secrets\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.597031 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.597007    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-bound-sa-token\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.597031 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.597027    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h228x\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-kube-api-access-h228x\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:17.629752 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:17.629727    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z"]
Jul 21 17:27:17.632397 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:17.632377    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod87803527_5457_4f07_9fa4_54080f6959a2.slice/crio-87fb1e4d0975dc3b3e08f06af670a95802a1fa319d81c092d7fdd0a5474d8c2c WatchSource:0}: Error finding container 87fb1e4d0975dc3b3e08f06af670a95802a1fa319d81c092d7fdd0a5474d8c2c: Status 404 returned error can't find the container with id 87fb1e4d0975dc3b3e08f06af670a95802a1fa319d81c092d7fdd0a5474d8c2c
Jul 21 17:27:18.079495 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:18.079462    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:18.079683 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:18.079600    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:27:18.079683 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:18.079617    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5574d67986-vk42j: secret "image-registry-tls" not found
Jul 21 17:27:18.079683 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:18.079668    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls podName:b31c70f4-3128-4aa3-b468-29c378079b26 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:19.079652354 +0000 UTC m=+98.637659440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls") pod "image-registry-5574d67986-vk42j" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26") : secret "image-registry-tls" not found
Jul 21 17:27:18.262653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:18.262613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z" event={"ID":"87803527-5457-4f07-9fa4-54080f6959a2","Type":"ContainerStarted","Data":"a0734fbe703ee4ec404372fada85b917206f8752cc9032157aee2ea43f669387"}
Jul 21 17:27:18.263144 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:18.262673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z" event={"ID":"87803527-5457-4f07-9fa4-54080f6959a2","Type":"ContainerStarted","Data":"87fb1e4d0975dc3b3e08f06af670a95802a1fa319d81c092d7fdd0a5474d8c2c"}
Jul 21 17:27:18.279723 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:18.279676    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-l5q5z" podStartSLOduration=1.279659626 podStartE2EDuration="1.279659626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:18.279131673 +0000 UTC m=+97.837138793" watchObservedRunningTime="2026-07-21 17:27:18.279659626 +0000 UTC m=+97.837666737"
Jul 21 17:27:18.281632 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:18.281610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:18.281761 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:18.281667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:18.281842 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:18.281769    2563 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:18.281842 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:18.281778    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:34.281762088 +0000 UTC m=+113.839769176 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : configmap references non-existent config key: service-ca.crt
Jul 21 17:27:18.282007 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:18.281870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:18.282007 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:18.281878    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls podName:b8e304a3-94a7-4671-a188-fbc216c91b6f nodeName:}" failed. No retries permitted until 2026-07-21 17:27:34.281862304 +0000 UTC m=+113.839869388 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-4fj4r" (UID: "b8e304a3-94a7-4671-a188-fbc216c91b6f") : secret "cluster-monitoring-operator-tls" not found
Jul 21 17:27:18.282007 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:18.281917    2563 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 21 17:27:18.282007 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:18.281953    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs podName:15e82759-7714-4b28-b4cb-c9289ae76056 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:34.281942032 +0000 UTC m=+113.839949118 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs") pod "router-default-7cfbdb577d-z6pf4" (UID: "15e82759-7714-4b28-b4cb-c9289ae76056") : secret "router-metrics-certs-default" not found
Jul 21 17:27:19.088588 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:19.088562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:19.088745 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:19.088712    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:27:19.088745 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:19.088727    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5574d67986-vk42j: secret "image-registry-tls" not found
Jul 21 17:27:19.088831 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:19.088779    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls podName:b31c70f4-3128-4aa3-b468-29c378079b26 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:21.088763485 +0000 UTC m=+100.646770570 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls") pod "image-registry-5574d67986-vk42j" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26") : secret "image-registry-tls" not found
Jul 21 17:27:19.796152 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:19.796055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:27:19.796591 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:19.796229    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 21 17:27:19.796591 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:19.796314    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls podName:e1a92e5b-3446-457c-bda1-e069f7fedc95 nodeName:}" failed. No retries permitted until 2026-07-21 17:28:23.796290692 +0000 UTC m=+163.354297783 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls") pod "dns-default-qmpzt" (UID: "e1a92e5b-3446-457c-bda1-e069f7fedc95") : secret "dns-default-metrics-tls" not found
Jul 21 17:27:21.105481 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:21.105436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:21.105976 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:21.105623    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 21 17:27:21.105976 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:21.105645    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5574d67986-vk42j: secret "image-registry-tls" not found
Jul 21 17:27:21.105976 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:21.105715    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls podName:b31c70f4-3128-4aa3-b468-29c378079b26 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:25.105693245 +0000 UTC m=+104.663700332 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls") pod "image-registry-5574d67986-vk42j" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26") : secret "image-registry-tls" not found
Jul 21 17:27:23.204489 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:23.204453    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-qq9ln"
Jul 21 17:27:23.423725 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:23.423680    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:23.423926 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:23.423809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:23.423966 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:23.423929    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:23.424000 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:23.423986    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls podName:1350fcd2-f330-46c1-a54d-06f482e0800a nodeName:}" failed. No retries permitted until 2026-07-21 17:27:39.423970313 +0000 UTC m=+118.981977400 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-dl229" (UID: "1350fcd2-f330-46c1-a54d-06f482e0800a") : secret "insights-runtime-extractor-tls" not found
Jul 21 17:27:23.424200 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:23.424180    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/1350fcd2-f330-46c1-a54d-06f482e0800a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:25.137464 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:25.137430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:25.139876 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:25.139856    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") pod \"image-registry-5574d67986-vk42j\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") " pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:25.222052 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:25.222024    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:25.379015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:25.378984    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5574d67986-vk42j"]
Jul 21 17:27:25.382349 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:25.382318    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb31c70f4_3128_4aa3_b468_29c378079b26.slice/crio-8b24894fd09e4df7caa310baead15f41edfb91bceb012af6b2167b2c0c44a068 WatchSource:0}: Error finding container 8b24894fd09e4df7caa310baead15f41edfb91bceb012af6b2167b2c0c44a068: Status 404 returned error can't find the container with id 8b24894fd09e4df7caa310baead15f41edfb91bceb012af6b2167b2c0c44a068
Jul 21 17:27:26.284562 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:26.284522    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5574d67986-vk42j" event={"ID":"b31c70f4-3128-4aa3-b468-29c378079b26","Type":"ContainerStarted","Data":"47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74"}
Jul 21 17:27:26.284562 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:26.284568    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5574d67986-vk42j" event={"ID":"b31c70f4-3128-4aa3-b468-29c378079b26","Type":"ContainerStarted","Data":"8b24894fd09e4df7caa310baead15f41edfb91bceb012af6b2167b2c0c44a068"}
Jul 21 17:27:26.285070 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:26.284599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:26.314660 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:26.314616    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5574d67986-vk42j" podStartSLOduration=9.314601395 podStartE2EDuration="9.314601395s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:26.314381459 +0000 UTC m=+105.872388566" watchObservedRunningTime="2026-07-21 17:27:26.314601395 +0000 UTC m=+105.872608502"
Jul 21 17:27:34.306261 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.306219    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:34.306670 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.306303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:34.306670 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.306349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:34.307807 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.307665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/15e82759-7714-4b28-b4cb-c9289ae76056-service-ca-bundle\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:34.309079 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.309052    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/15e82759-7714-4b28-b4cb-c9289ae76056-metrics-certs\") pod \"router-default-7cfbdb577d-z6pf4\" (UID: \"15e82759-7714-4b28-b4cb-c9289ae76056\") " pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:34.312400 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.312375    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/b8e304a3-94a7-4671-a188-fbc216c91b6f-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-4fj4r\" (UID: \"b8e304a3-94a7-4671-a188-fbc216c91b6f\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:34.314091 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.314074    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"
Jul 21 17:27:34.319669 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.319648    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:34.449569 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.449540    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r"]
Jul 21 17:27:34.453086 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:34.453053    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb8e304a3_94a7_4671_a188_fbc216c91b6f.slice/crio-aa26795f6a0365b85b4a5fa4c66238d7eadc6304b93adc7bb3cf4629006ed5e4 WatchSource:0}: Error finding container aa26795f6a0365b85b4a5fa4c66238d7eadc6304b93adc7bb3cf4629006ed5e4: Status 404 returned error can't find the container with id aa26795f6a0365b85b4a5fa4c66238d7eadc6304b93adc7bb3cf4629006ed5e4
Jul 21 17:27:34.471275 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:34.471250    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-7cfbdb577d-z6pf4"]
Jul 21 17:27:34.475016 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:34.474985    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e82759_7714_4b28_b4cb_c9289ae76056.slice/crio-a3ec597f75c61c9e8c9d34c4dacdc0ef3b750197179a64f2a30d8faa5ba77456 WatchSource:0}: Error finding container a3ec597f75c61c9e8c9d34c4dacdc0ef3b750197179a64f2a30d8faa5ba77456: Status 404 returned error can't find the container with id a3ec597f75c61c9e8c9d34c4dacdc0ef3b750197179a64f2a30d8faa5ba77456
Jul 21 17:27:35.316390 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:35.316356    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r" event={"ID":"b8e304a3-94a7-4671-a188-fbc216c91b6f","Type":"ContainerStarted","Data":"aa26795f6a0365b85b4a5fa4c66238d7eadc6304b93adc7bb3cf4629006ed5e4"}
Jul 21 17:27:35.318132 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:35.318089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4" event={"ID":"15e82759-7714-4b28-b4cb-c9289ae76056","Type":"ContainerStarted","Data":"8826cbd26f2ca53c376c258d4758d4ca9bed1523090eafa14ea255e1dd1f04f4"}
Jul 21 17:27:35.318288 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:35.318139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4" event={"ID":"15e82759-7714-4b28-b4cb-c9289ae76056","Type":"ContainerStarted","Data":"a3ec597f75c61c9e8c9d34c4dacdc0ef3b750197179a64f2a30d8faa5ba77456"}
Jul 21 17:27:35.320537 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:35.320505    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:35.323196 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:35.323174    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:35.344075 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:35.344030    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4" podStartSLOduration=33.344015608 podStartE2EDuration="33.344015608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:35.342665527 +0000 UTC m=+114.900672653" watchObservedRunningTime="2026-07-21 17:27:35.344015608 +0000 UTC m=+114.902022693"
Jul 21 17:27:36.323654 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:36.323588    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r" event={"ID":"b8e304a3-94a7-4671-a188-fbc216c91b6f","Type":"ContainerStarted","Data":"e2d7db17e42264487cbfc8cd5a1e28079684db939e7b99776ae5002a46105c07"}
Jul 21 17:27:36.324167 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:36.324037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:36.325429 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:36.325404    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-7cfbdb577d-z6pf4"
Jul 21 17:27:36.346375 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:36.346330    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-4fj4r" podStartSLOduration=32.612757844 podStartE2EDuration="34.34631563s" totalImagesPullingTime="1.733557786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:36.344643731 +0000 UTC m=+115.902650838" watchObservedRunningTime="2026-07-21 17:27:36.34631563 +0000 UTC m=+115.904322736"
Jul 21 17:27:38.088132 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.088098    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5574d67986-vk42j"]
Jul 21 17:27:38.197525 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.197493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"]
Jul 21 17:27:38.199348 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.199333    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"
Jul 21 17:27:38.203243 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.203224    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-lzdxl\""
Jul 21 17:27:38.203493 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.203478    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 21 17:27:38.203821 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.203781    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 21 17:27:38.207940 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.207922    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"]
Jul 21 17:27:38.210192 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.210166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"
Jul 21 17:27:38.213619 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.213600    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 21 17:27:38.213760 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.213749    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-lzblt\""
Jul 21 17:27:38.257956 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.257926    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"]
Jul 21 17:27:38.258765 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.258730    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"]
Jul 21 17:27:38.336290 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.336259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/62bde93c-82cd-4ab9-8a93-b22780216f0f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-8wlkh\" (UID: \"62bde93c-82cd-4ab9-8a93-b22780216f0f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"
Jul 21 17:27:38.336435 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.336296    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/699ff256-14c0-4378-8721-9a322e2dc0ec-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-dp25d\" (UID: \"699ff256-14c0-4378-8721-9a322e2dc0ec\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"
Jul 21 17:27:38.336435 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.336413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/699ff256-14c0-4378-8721-9a322e2dc0ec-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-dp25d\" (UID: \"699ff256-14c0-4378-8721-9a322e2dc0ec\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"
Jul 21 17:27:38.437643 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.437559    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/62bde93c-82cd-4ab9-8a93-b22780216f0f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-8wlkh\" (UID: \"62bde93c-82cd-4ab9-8a93-b22780216f0f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"
Jul 21 17:27:38.437805 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.437703    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/699ff256-14c0-4378-8721-9a322e2dc0ec-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-dp25d\" (UID: \"699ff256-14c0-4378-8721-9a322e2dc0ec\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"
Jul 21 17:27:38.438018 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.437995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/699ff256-14c0-4378-8721-9a322e2dc0ec-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-dp25d\" (UID: \"699ff256-14c0-4378-8721-9a322e2dc0ec\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"
Jul 21 17:27:38.438411 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.438381    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/699ff256-14c0-4378-8721-9a322e2dc0ec-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-dp25d\" (UID: \"699ff256-14c0-4378-8721-9a322e2dc0ec\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"
Jul 21 17:27:38.440281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.440259    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/62bde93c-82cd-4ab9-8a93-b22780216f0f-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-8wlkh\" (UID: \"62bde93c-82cd-4ab9-8a93-b22780216f0f\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"
Jul 21 17:27:38.440412 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.440393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/699ff256-14c0-4378-8721-9a322e2dc0ec-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-dp25d\" (UID: \"699ff256-14c0-4378-8721-9a322e2dc0ec\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"
Jul 21 17:27:38.508420 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.508392    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"
Jul 21 17:27:38.520023 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.520000    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"
Jul 21 17:27:38.659632 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.659587    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d"]
Jul 21 17:27:38.662670 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:38.662641    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod699ff256_14c0_4378_8721_9a322e2dc0ec.slice/crio-631fbb73f68b34a4676cb5451678266e1e03c8a9da1915370abc926a83f1de0d WatchSource:0}: Error finding container 631fbb73f68b34a4676cb5451678266e1e03c8a9da1915370abc926a83f1de0d: Status 404 returned error can't find the container with id 631fbb73f68b34a4676cb5451678266e1e03c8a9da1915370abc926a83f1de0d
Jul 21 17:27:38.682636 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:38.682614    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"]
Jul 21 17:27:38.685466 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:38.685445    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62bde93c_82cd_4ab9_8a93_b22780216f0f.slice/crio-8492875418de9fd7515ac1ab0d6c58622ecd990e1f590acaf34db423a57953f6 WatchSource:0}: Error finding container 8492875418de9fd7515ac1ab0d6c58622ecd990e1f590acaf34db423a57953f6: Status 404 returned error can't find the container with id 8492875418de9fd7515ac1ab0d6c58622ecd990e1f590acaf34db423a57953f6
Jul 21 17:27:39.334864 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:39.334818    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh" event={"ID":"62bde93c-82cd-4ab9-8a93-b22780216f0f","Type":"ContainerStarted","Data":"8492875418de9fd7515ac1ab0d6c58622ecd990e1f590acaf34db423a57953f6"}
Jul 21 17:27:39.336062 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:39.336030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d" event={"ID":"699ff256-14c0-4378-8721-9a322e2dc0ec","Type":"ContainerStarted","Data":"631fbb73f68b34a4676cb5451678266e1e03c8a9da1915370abc926a83f1de0d"}
Jul 21 17:27:39.447908 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:39.447872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:39.450475 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:39.450450    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/1350fcd2-f330-46c1-a54d-06f482e0800a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-dl229\" (UID: \"1350fcd2-f330-46c1-a54d-06f482e0800a\") " pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:39.472537 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:39.472509    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-dl229"
Jul 21 17:27:40.082657 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.082629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-dl229"]
Jul 21 17:27:40.085704 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:40.085678    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1350fcd2_f330_46c1_a54d_06f482e0800a.slice/crio-efde4ca8d8186dd9c2b4530465ff74079361ad8e98978c694a0f9806ccd1a9c5 WatchSource:0}: Error finding container efde4ca8d8186dd9c2b4530465ff74079361ad8e98978c694a0f9806ccd1a9c5: Status 404 returned error can't find the container with id efde4ca8d8186dd9c2b4530465ff74079361ad8e98978c694a0f9806ccd1a9c5
Jul 21 17:27:40.340610 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.340576    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl229" event={"ID":"1350fcd2-f330-46c1-a54d-06f482e0800a","Type":"ContainerStarted","Data":"efde4ca8d8186dd9c2b4530465ff74079361ad8e98978c694a0f9806ccd1a9c5"}
Jul 21 17:27:40.341994 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.341960    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh" event={"ID":"62bde93c-82cd-4ab9-8a93-b22780216f0f","Type":"ContainerStarted","Data":"1bd967e103fd2a5e174ea1a0ddadc75f2c275e0401f36e6a444589a89c107e13"}
Jul 21 17:27:40.342165 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.342144    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"
Jul 21 17:27:40.343466 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.343443    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d" event={"ID":"699ff256-14c0-4378-8721-9a322e2dc0ec","Type":"ContainerStarted","Data":"29eaf1bcbcf5c1db87543f5984ebaa81daefad1e993e2817cb4e17a76a21dcbf"}
Jul 21 17:27:40.347168 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.347150    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh"
Jul 21 17:27:40.388000 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.387957    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-8wlkh" podStartSLOduration=1.128367353 podStartE2EDuration="2.387939386s" totalImagesPullingTime="1.259572033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:40.363160825 +0000 UTC m=+119.921167933" watchObservedRunningTime="2026-07-21 17:27:40.387939386 +0000 UTC m=+119.945946493"
Jul 21 17:27:40.388263 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.388241    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-dp25d" podStartSLOduration=1.107627878 podStartE2EDuration="2.388235662s" totalImagesPullingTime="1.280607784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:40.387176183 +0000 UTC m=+119.945183291" watchObservedRunningTime="2026-07-21 17:27:40.388235662 +0000 UTC m=+119.946242769"
Jul 21 17:27:40.814052 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.814019    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"]
Jul 21 17:27:40.817337 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.817306    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:40.820273 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.820243    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 21 17:27:40.820400 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.820282    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-xqkr8\""
Jul 21 17:27:40.821510 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.821488    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 21 17:27:40.821611 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.821541    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 21 17:27:40.830506 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.830434    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"]
Jul 21 17:27:40.960066 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.960026    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:40.960253 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.960078    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:40.960253 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.960146    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:40.960253 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:40.960242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l995t\" (UniqueName: \"kubernetes.io/projected/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-kube-api-access-l995t\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.061398 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.061362    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l995t\" (UniqueName: \"kubernetes.io/projected/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-kube-api-access-l995t\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.061573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.061409    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.061573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.061434    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.061573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.061473    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.062347 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.062322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.064528 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.064474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.064602 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.064573    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.072040 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.072018    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l995t\" (UniqueName: \"kubernetes.io/projected/cdf7bfbf-2443-411f-af32-4c6f2ee61b65-kube-api-access-l995t\") pod \"prometheus-operator-5945b5cfd-bhsbb\" (UID: \"cdf7bfbf-2443-411f-af32-4c6f2ee61b65\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.134650 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.134627    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"
Jul 21 17:27:41.289110 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.289080    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb"]
Jul 21 17:27:41.292689 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:41.292650    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcdf7bfbf_2443_411f_af32_4c6f2ee61b65.slice/crio-ee16c11dc45228be8804362c94b33bf9db4e8bfb1ed3a6219946ef50497c1d03 WatchSource:0}: Error finding container ee16c11dc45228be8804362c94b33bf9db4e8bfb1ed3a6219946ef50497c1d03: Status 404 returned error can't find the container with id ee16c11dc45228be8804362c94b33bf9db4e8bfb1ed3a6219946ef50497c1d03
Jul 21 17:27:41.349144 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:41.349051    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb" event={"ID":"cdf7bfbf-2443-411f-af32-4c6f2ee61b65","Type":"ContainerStarted","Data":"ee16c11dc45228be8804362c94b33bf9db4e8bfb1ed3a6219946ef50497c1d03"}
Jul 21 17:27:45.367361 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:45.367318    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl229" event={"ID":"1350fcd2-f330-46c1-a54d-06f482e0800a","Type":"ContainerStarted","Data":"2fbb80a160275329c3e9748509ab028cdbbcfa85899c5b1632cd772dfcec7762"}
Jul 21 17:27:45.369598 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:45.369563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb" event={"ID":"cdf7bfbf-2443-411f-af32-4c6f2ee61b65","Type":"ContainerStarted","Data":"f5a1aa637e2ee456360c410d2cba38d663f4ef4b75b6d241c6f98dffe9a79836"}
Jul 21 17:27:45.369725 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:45.369624    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb" event={"ID":"cdf7bfbf-2443-411f-af32-4c6f2ee61b65","Type":"ContainerStarted","Data":"ec7a2efdccf74aa55d3ec09a3cd6ad40cfba3f731336ebe111ebcb4674d28f36"}
Jul 21 17:27:45.391544 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:45.391433    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-bhsbb" podStartSLOduration=1.547702749 podStartE2EDuration="5.391414048s" totalImagesPullingTime="3.843711299s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:45.389460515 +0000 UTC m=+124.947467647" watchObservedRunningTime="2026-07-21 17:27:45.391414048 +0000 UTC m=+124.949421158"
Jul 21 17:27:47.305802 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.305749    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"]
Jul 21 17:27:47.318453 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.318423    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.324968 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.324114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-5hrwm\""
Jul 21 17:27:47.324968 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.324703    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Jul 21 17:27:47.324968 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.324947    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Jul 21 17:27:47.325422 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.325399    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"]
Jul 21 17:27:47.346240 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.346214    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"]
Jul 21 17:27:47.351191 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.351168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.357076 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.357052    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 21 17:27:47.357338 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.357308    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 21 17:27:47.357459 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.357443    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-sppwt\""
Jul 21 17:27:47.358394 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.357780    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 21 17:27:47.373171 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.367348    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-4z6f4"]
Jul 21 17:27:47.390375 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.379037    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"]
Jul 21 17:27:47.390375 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.379168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.390375 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.386985    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 21 17:27:47.390375 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.387286    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t76wx\""
Jul 21 17:27:47.390375 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.387489    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 21 17:27:47.390375 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.387765    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 21 17:27:47.413808 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.413749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wddbl\" (UniqueName: \"kubernetes.io/projected/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-api-access-wddbl\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.413808 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.413804    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.414042 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.413836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dad95311-bc9c-41dc-88b7-637efe0ff6f9-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.414042 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.413890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-wtmp\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.414042 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.413949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dad95311-bc9c-41dc-88b7-637efe0ff6f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.414042 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.413987    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lwps8\" (UniqueName: \"kubernetes.io/projected/dad95311-bc9c-41dc-88b7-637efe0ff6f9-kube-api-access-lwps8\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.414042 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-textfile\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.414318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414051    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.414318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414110    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.414318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414140    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.414318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414178    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5q6xh\" (UniqueName: \"kubernetes.io/projected/51b3d631-7caa-471a-a098-4fa381ed02d7-kube-api-access-5q6xh\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.414318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414197    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b3d631-7caa-471a-a098-4fa381ed02d7-metrics-client-ca\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.414318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414240    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.414318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.414678 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414327    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.414678 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-root\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.414678 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad95311-bc9c-41dc-88b7-637efe0ff6f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.414678 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-sys\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.414678 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.414439    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-tls\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.515029 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.514980    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dad95311-bc9c-41dc-88b7-637efe0ff6f9-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.515029 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-wtmp\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.515289 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515062    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dad95311-bc9c-41dc-88b7-637efe0ff6f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.515289 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lwps8\" (UniqueName: \"kubernetes.io/projected/dad95311-bc9c-41dc-88b7-637efe0ff6f9-kube-api-access-lwps8\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.515289 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515140    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-textfile\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.515289 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.515289 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.515289 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515241    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.515289 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515288    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5q6xh\" (UniqueName: \"kubernetes.io/projected/51b3d631-7caa-471a-a098-4fa381ed02d7-kube-api-access-5q6xh\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.515653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b3d631-7caa-471a-a098-4fa381ed02d7-metrics-client-ca\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.515653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515343    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.515653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.515653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515412    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.515653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515469    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-root\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.515653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515492    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad95311-bc9c-41dc-88b7-637efe0ff6f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.515653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515513    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-textfile\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.516011 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515747    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dad95311-bc9c-41dc-88b7-637efe0ff6f9-metrics-client-ca\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.516011 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:47.515993    2563 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 21 17:27:47.516099 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:47.516072    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-tls podName:40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae nodeName:}" failed. No retries permitted until 2026-07-21 17:27:48.016052281 +0000 UTC m=+127.574059384 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-tls") pod "kube-state-metrics-56c5594674-cvxb8" (UID: "40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae") : secret "kube-state-metrics-tls" not found
Jul 21 17:27:47.516099 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516091    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-accelerators-collector-config\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.516204 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516146    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-sys\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.516204 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516196    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-root\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.516300 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/51b3d631-7caa-471a-a098-4fa381ed02d7-metrics-client-ca\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.516300 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516274    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.516300 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-wtmp\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.516570 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.515521    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/51b3d631-7caa-471a-a098-4fa381ed02d7-sys\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.516643 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516599    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-tls\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.516643 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wddbl\" (UniqueName: \"kubernetes.io/projected/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-api-access-wddbl\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.516758 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.516758 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516667    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.516925 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.516900    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.519280 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.519227    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.519386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.519336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dad95311-bc9c-41dc-88b7-637efe0ff6f9-openshift-state-metrics-tls\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.519555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.519524    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/51b3d631-7caa-471a-a098-4fa381ed02d7-node-exporter-tls\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.519678 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.519650    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.520471 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.520443    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dad95311-bc9c-41dc-88b7-637efe0ff6f9-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.525600 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.525580    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wddbl\" (UniqueName: \"kubernetes.io/projected/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-api-access-wddbl\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:47.525715 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.525689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5q6xh\" (UniqueName: \"kubernetes.io/projected/51b3d631-7caa-471a-a098-4fa381ed02d7-kube-api-access-5q6xh\") pod \"node-exporter-4z6f4\" (UID: \"51b3d631-7caa-471a-a098-4fa381ed02d7\") " pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:47.526582 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.526564    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lwps8\" (UniqueName: \"kubernetes.io/projected/dad95311-bc9c-41dc-88b7-637efe0ff6f9-kube-api-access-lwps8\") pod \"openshift-state-metrics-79f6958cdf-l4rs5\" (UID: \"dad95311-bc9c-41dc-88b7-637efe0ff6f9\") " pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.630244 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.630156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"
Jul 21 17:27:47.694425 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:47.694390    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-4z6f4"
Jul 21 17:27:48.020173 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.020132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:48.022851 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.022827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-cvxb8\" (UID: \"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:48.094324 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.094297    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:27:48.268568 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.268527    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:27:48.269459 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.269428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"
Jul 21 17:27:48.280012 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.279991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.283262 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.283235    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 21 17:27:48.283385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.283324    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 21 17:27:48.283475 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.283460    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 21 17:27:48.283543 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.283530    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z9cgb\""
Jul 21 17:27:48.284147 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.283723    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 21 17:27:48.284147 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.283836    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 21 17:27:48.284147 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.283951    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 21 17:27:48.284147 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.283963    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 21 17:27:48.284147 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.284048    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 21 17:27:48.284465 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.284235    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 21 17:27:48.290924 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.290899    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:27:48.326698 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.326664    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.326709    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-out\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.326735    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.326820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.326867    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.326905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.326949    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5j8h\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-kube-api-access-f5j8h\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.326975    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.327022    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327431 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.327090    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-volume\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327431 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.327127    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-web-config\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327431 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.327173    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.327431 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.327205    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428104 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5j8h\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-kube-api-access-f5j8h\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428104 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-volume\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428196    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-web-config\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428232    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428296    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-out\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428347    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428827 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428371    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428827 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.428392    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.428827 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:48.428562    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle podName:32aeeb27-734e-431a-b23e-0b837e5e8e75 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:48.928540472 +0000 UTC m=+128.486547561 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75") : configmap references non-existent config key: ca-bundle.crt
Jul 21 17:27:48.428827 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:48.428813    2563 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Jul 21 17:27:48.429029 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:27:48.428900    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls podName:32aeeb27-734e-431a-b23e-0b837e5e8e75 nodeName:}" failed. No retries permitted until 2026-07-21 17:27:48.928886907 +0000 UTC m=+128.486893992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75") : secret "alertmanager-main-tls" not found
Jul 21 17:27:48.429086 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.429040    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.430895 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.430470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.431638 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.431551    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.431833 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.431662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.431833 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.431763    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-volume\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.432565 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.432519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-tls-assets\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.433531 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.433337    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.433531 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.433342    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-out\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.434419 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.434393    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.434512 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.434457    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-web-config\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.451482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.451463    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5j8h\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-kube-api-access-f5j8h\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.762765 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:48.762721    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51b3d631_7caa_471a_a098_4fa381ed02d7.slice/crio-92503347d1a911ec2d849c100c3310cd09fe7441d053b2041ca9cdcf0a05da81 WatchSource:0}: Error finding container 92503347d1a911ec2d849c100c3310cd09fe7441d053b2041ca9cdcf0a05da81: Status 404 returned error can't find the container with id 92503347d1a911ec2d849c100c3310cd09fe7441d053b2041ca9cdcf0a05da81
Jul 21 17:27:48.908231 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:48.908202    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40bc3b96_f384_4cac_89e9_ed1d3fb3f9ae.slice/crio-e796c4f3b2a4ddeca9b5fc1cd1f7c35a9c3ba6fc28ece7e4dd17d9c3cc22e74d WatchSource:0}: Error finding container e796c4f3b2a4ddeca9b5fc1cd1f7c35a9c3ba6fc28ece7e4dd17d9c3cc22e74d: Status 404 returned error can't find the container with id e796c4f3b2a4ddeca9b5fc1cd1f7c35a9c3ba6fc28ece7e4dd17d9c3cc22e74d
Jul 21 17:27:48.909635 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.909614    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-cvxb8"]
Jul 21 17:27:48.924703 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.924666    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5"]
Jul 21 17:27:48.926854 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:48.926828    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddad95311_bc9c_41dc_88b7_637efe0ff6f9.slice/crio-54f1b84fc9fd54d450f338dc9eac295bd97da316bcadff7684a5d2c9266f5300 WatchSource:0}: Error finding container 54f1b84fc9fd54d450f338dc9eac295bd97da316bcadff7684a5d2c9266f5300: Status 404 returned error can't find the container with id 54f1b84fc9fd54d450f338dc9eac295bd97da316bcadff7684a5d2c9266f5300
Jul 21 17:27:48.932698 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.932678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.932768 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.932734    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.933684 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.933662    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:48.935273 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:48.935238    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:49.195600 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.195567    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:27:49.381941 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.381906    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:27:49.387045 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:49.387007    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32aeeb27_734e_431a_b23e_0b837e5e8e75.slice/crio-1e560aeec0fde756d5f5a8bd0f1052ebe9c0e7dbd54f9ffb4ee1acd5767a9172 WatchSource:0}: Error finding container 1e560aeec0fde756d5f5a8bd0f1052ebe9c0e7dbd54f9ffb4ee1acd5767a9172: Status 404 returned error can't find the container with id 1e560aeec0fde756d5f5a8bd0f1052ebe9c0e7dbd54f9ffb4ee1acd5767a9172
Jul 21 17:27:49.405087 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.405004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8" event={"ID":"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae","Type":"ContainerStarted","Data":"e796c4f3b2a4ddeca9b5fc1cd1f7c35a9c3ba6fc28ece7e4dd17d9c3cc22e74d"}
Jul 21 17:27:49.406861 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.406732    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4z6f4" event={"ID":"51b3d631-7caa-471a-a098-4fa381ed02d7","Type":"ContainerStarted","Data":"92503347d1a911ec2d849c100c3310cd09fe7441d053b2041ca9cdcf0a05da81"}
Jul 21 17:27:49.409851 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.409775    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5" event={"ID":"dad95311-bc9c-41dc-88b7-637efe0ff6f9","Type":"ContainerStarted","Data":"dbad8cc0e1c1029d75192b947ace1c38e731eda2180492e2eb0f0a7a78c5f253"}
Jul 21 17:27:49.409957 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.409907    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5" event={"ID":"dad95311-bc9c-41dc-88b7-637efe0ff6f9","Type":"ContainerStarted","Data":"7f6f83fd2ee0687daeae7cbefaed5d47ce63815d783622eef947a0d4338f4ead"}
Jul 21 17:27:49.409957 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.409939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5" event={"ID":"dad95311-bc9c-41dc-88b7-637efe0ff6f9","Type":"ContainerStarted","Data":"54f1b84fc9fd54d450f338dc9eac295bd97da316bcadff7684a5d2c9266f5300"}
Jul 21 17:27:49.413234 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.413209    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl229" event={"ID":"1350fcd2-f330-46c1-a54d-06f482e0800a","Type":"ContainerStarted","Data":"db274116c4e2c7be4172917af862d9431562649a0389d6a53d623c2eea3cb2b9"}
Jul 21 17:27:49.414604 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:49.414570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerStarted","Data":"1e560aeec0fde756d5f5a8bd0f1052ebe9c0e7dbd54f9ffb4ee1acd5767a9172"}
Jul 21 17:27:50.423000 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:50.422918    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8" event={"ID":"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae","Type":"ContainerStarted","Data":"e9a8defe8d1e87e5de001a20b21ea0e15dcf8da2fb238f5dbe753b3e7cfd9c3c"}
Jul 21 17:27:50.423652 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:50.423024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8" event={"ID":"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae","Type":"ContainerStarted","Data":"d2c7b386e37945d6007f091884b95a6c25b42ebf14806b7330f72f746a70a2c6"}
Jul 21 17:27:50.430433 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:50.429630    2563 generic.go:356] "Generic (PLEG): container finished" podID="51b3d631-7caa-471a-a098-4fa381ed02d7" containerID="b0dddbbe349422dc4ff79aaf154a04e4fb17b4d321dee2605c4cdcbd09b18dfb" exitCode=0
Jul 21 17:27:50.430433 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:50.429716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4z6f4" event={"ID":"51b3d631-7caa-471a-a098-4fa381ed02d7","Type":"ContainerDied","Data":"b0dddbbe349422dc4ff79aaf154a04e4fb17b4d321dee2605c4cdcbd09b18dfb"}
Jul 21 17:27:50.754063 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:50.753436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:27:50.759207 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:50.759105    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfe68ed-2ecc-4369-adcd-235f6b199b24-metrics-certs\") pod \"network-metrics-daemon-rz8hm\" (UID: \"8cfe68ed-2ecc-4369-adcd-235f6b199b24\") " pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:27:51.033820 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.033724    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-swxfv\""
Jul 21 17:27:51.040597 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.040365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-rz8hm"
Jul 21 17:27:51.436943 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.436861    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8" event={"ID":"40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae","Type":"ContainerStarted","Data":"5152f8929ec55cd66f63920a6af5d0f0ca33433a9b359b855de446c594e6b671"}
Jul 21 17:27:51.439268 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.439244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4z6f4" event={"ID":"51b3d631-7caa-471a-a098-4fa381ed02d7","Type":"ContainerStarted","Data":"d1816962e512a156123be1c616febea1df4d27431466e95a022d0e4441e96a5f"}
Jul 21 17:27:51.439451 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.439292    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-4z6f4" event={"ID":"51b3d631-7caa-471a-a098-4fa381ed02d7","Type":"ContainerStarted","Data":"378951370b98d29182b8b02469d03ed3d0b274bc02bb53176f7e1c5591f08ddb"}
Jul 21 17:27:51.463113 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.463052    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-cvxb8" podStartSLOduration=3.173254293 podStartE2EDuration="4.463034765s" totalImagesPullingTime="1.289780472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:51.462723921 +0000 UTC m=+131.020731052" watchObservedRunningTime="2026-07-21 17:27:51.463034765 +0000 UTC m=+131.021041854"
Jul 21 17:27:51.487313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.487266    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-4z6f4" podStartSLOduration=2.9972938 podStartE2EDuration="4.487250277s" totalImagesPullingTime="1.386065198s" totalInitContainerRuntime="103.891279ms" isStatefulPod=true podCreationTimestamp="2026-07-21 17:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:51.486228936 +0000 UTC m=+131.044236042" watchObservedRunningTime="2026-07-21 17:27:51.487250277 +0000 UTC m=+131.045257386"
Jul 21 17:27:51.804051 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.804017    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"]
Jul 21 17:27:51.814941 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.814910    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.816032 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.816010    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"]
Jul 21 17:27:51.818303 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.818262    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 21 17:27:51.818829 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.818513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 21 17:27:51.820884 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.820853    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-529mu4e5bh1l1\""
Jul 21 17:27:51.820884 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.820879    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 21 17:27:51.821261 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.821239    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 21 17:27:51.821964 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.821943    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-85m2j\""
Jul 21 17:27:51.867573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.865200    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/791f4d17-aa77-403b-a5a5-0314c63a72c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.867573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.865276    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-secret-metrics-server-client-certs\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.867573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.865334    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/791f4d17-aa77-403b-a5a5-0314c63a72c7-metrics-server-audit-profiles\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.867573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.865368    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-secret-metrics-server-tls\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.867573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.865423    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-client-ca-bundle\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.867573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.865473    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smghj\" (UniqueName: \"kubernetes.io/projected/791f4d17-aa77-403b-a5a5-0314c63a72c7-kube-api-access-smghj\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.867573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.865530    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/791f4d17-aa77-403b-a5a5-0314c63a72c7-audit-log\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.966159 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.966113    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-secret-metrics-server-client-certs\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.966327 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.966172    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/791f4d17-aa77-403b-a5a5-0314c63a72c7-metrics-server-audit-profiles\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.966327 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.966202    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-secret-metrics-server-tls\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.966327 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.966249    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-client-ca-bundle\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.966327 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.966290    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smghj\" (UniqueName: \"kubernetes.io/projected/791f4d17-aa77-403b-a5a5-0314c63a72c7-kube-api-access-smghj\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.966544 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.966455    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/791f4d17-aa77-403b-a5a5-0314c63a72c7-audit-log\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.966544 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.966513    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/791f4d17-aa77-403b-a5a5-0314c63a72c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.966909 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.966878    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/791f4d17-aa77-403b-a5a5-0314c63a72c7-audit-log\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.967472 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.967416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/791f4d17-aa77-403b-a5a5-0314c63a72c7-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.967679 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.967653    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/791f4d17-aa77-403b-a5a5-0314c63a72c7-metrics-server-audit-profiles\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.969464 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.969440    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-secret-metrics-server-client-certs\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.969565 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.969523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-secret-metrics-server-tls\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.969998 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.969956    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/791f4d17-aa77-403b-a5a5-0314c63a72c7-client-ca-bundle\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:51.978747 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:51.978722    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smghj\" (UniqueName: \"kubernetes.io/projected/791f4d17-aa77-403b-a5a5-0314c63a72c7-kube-api-access-smghj\") pod \"metrics-server-7bc5f9ff5d-8fnrd\" (UID: \"791f4d17-aa77-403b-a5a5-0314c63a72c7\") " pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:52.128128 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.128042    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:27:52.514490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.514447    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7759f6578c-bnrrl"]
Jul 21 17:27:52.525517 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.525493    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.528390 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.528360    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 21 17:27:52.528390 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.528376    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 21 17:27:52.528390 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.528361    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-877ws\""
Jul 21 17:27:52.528573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.528505    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 21 17:27:52.528573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.528531    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 21 17:27:52.529129 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.529106    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 21 17:27:52.532621 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.532570    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7759f6578c-bnrrl"]
Jul 21 17:27:52.536536 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.536513    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 21 17:27:52.570844 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.570812    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-telemeter-client-tls\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.570981 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.570858    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-serving-certs-ca-bundle\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.570981 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.570877    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-metrics-client-ca\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.570981 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.570935    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-secret-telemeter-client\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.570981 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.570973    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.571153 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.571016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-federate-client-tls\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.571153 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.571034    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7vh4\" (UniqueName: \"kubernetes.io/projected/7b85463b-9b67-435a-96d7-d7f0f52f0782-kube-api-access-x7vh4\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.571153 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.571054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.672218 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.672179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-secret-telemeter-client\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.672411 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.672245    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.672807 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.672287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-federate-client-tls\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.672951 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.672853    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7vh4\" (UniqueName: \"kubernetes.io/projected/7b85463b-9b67-435a-96d7-d7f0f52f0782-kube-api-access-x7vh4\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.672951 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.672890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.673066 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.672991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-telemeter-client-tls\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.673066 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.673033    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-serving-certs-ca-bundle\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.673066 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.673055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-metrics-client-ca\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.673829 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.673800    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-metrics-client-ca\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.674115 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.674089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.674313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.674294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7b85463b-9b67-435a-96d7-d7f0f52f0782-serving-certs-ca-bundle\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.676168 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.676128    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-telemeter-client-tls\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.676284 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.676263    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-secret-telemeter-client\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.676647 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.676628    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.676719 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.676699    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/7b85463b-9b67-435a-96d7-d7f0f52f0782-federate-client-tls\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.696343 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.696315    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7vh4\" (UniqueName: \"kubernetes.io/projected/7b85463b-9b67-435a-96d7-d7f0f52f0782-kube-api-access-x7vh4\") pod \"telemeter-client-7759f6578c-bnrrl\" (UID: \"7b85463b-9b67-435a-96d7-d7f0f52f0782\") " pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:52.838201 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:52.838121    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl"
Jul 21 17:27:53.263698 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.263661    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-rz8hm"]
Jul 21 17:27:53.270445 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:53.270306    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfe68ed_2ecc_4369_adcd_235f6b199b24.slice/crio-a8a116ba1d4941884203132345398037fd5491562af89ce1f3096c8fced066d5 WatchSource:0}: Error finding container a8a116ba1d4941884203132345398037fd5491562af89ce1f3096c8fced066d5: Status 404 returned error can't find the container with id a8a116ba1d4941884203132345398037fd5491562af89ce1f3096c8fced066d5
Jul 21 17:27:53.286105 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.286078    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"]
Jul 21 17:27:53.292213 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.292190    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7759f6578c-bnrrl"]
Jul 21 17:27:53.311424 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:53.311387    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b85463b_9b67_435a_96d7_d7f0f52f0782.slice/crio-71fcac2f35587bde107b44282657546969963b1ff543ca861113f6640e93aa61 WatchSource:0}: Error finding container 71fcac2f35587bde107b44282657546969963b1ff543ca861113f6640e93aa61: Status 404 returned error can't find the container with id 71fcac2f35587bde107b44282657546969963b1ff543ca861113f6640e93aa61
Jul 21 17:27:53.311675 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:53.311650    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod791f4d17_aa77_403b_a5a5_0314c63a72c7.slice/crio-a083cf761d518b73701276878e99105ac47cf698514bc8a134dd5935e9c827fa WatchSource:0}: Error finding container a083cf761d518b73701276878e99105ac47cf698514bc8a134dd5935e9c827fa: Status 404 returned error can't find the container with id a083cf761d518b73701276878e99105ac47cf698514bc8a134dd5935e9c827fa
Jul 21 17:27:53.448704 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.448613    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl" event={"ID":"7b85463b-9b67-435a-96d7-d7f0f52f0782","Type":"ContainerStarted","Data":"71fcac2f35587bde107b44282657546969963b1ff543ca861113f6640e93aa61"}
Jul 21 17:27:53.451331 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.451297    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-dl229" event={"ID":"1350fcd2-f330-46c1-a54d-06f482e0800a","Type":"ContainerStarted","Data":"3e27d8089d850fd02e2d72d7fd9980433f994475dcddb2118700b2cad49e6a22"}
Jul 21 17:27:53.454588 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.454556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5" event={"ID":"dad95311-bc9c-41dc-88b7-637efe0ff6f9","Type":"ContainerStarted","Data":"a97238cbfc499add12094d1c107a2ad8ba6a8d5ba4f44b760f685f8218e3b03e"}
Jul 21 17:27:53.456148 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.456118    2563 generic.go:356] "Generic (PLEG): container finished" podID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerID="28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517" exitCode=0
Jul 21 17:27:53.456238 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.456203    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerDied","Data":"28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517"}
Jul 21 17:27:53.458777 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.458733    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd" event={"ID":"791f4d17-aa77-403b-a5a5-0314c63a72c7","Type":"ContainerStarted","Data":"a083cf761d518b73701276878e99105ac47cf698514bc8a134dd5935e9c827fa"}
Jul 21 17:27:53.460032 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.459988    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rz8hm" event={"ID":"8cfe68ed-2ecc-4369-adcd-235f6b199b24","Type":"ContainerStarted","Data":"a8a116ba1d4941884203132345398037fd5491562af89ce1f3096c8fced066d5"}
Jul 21 17:27:53.478654 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.478604    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-dl229" podStartSLOduration=33.628128103 podStartE2EDuration="46.478587848s" totalImagesPullingTime="12.850459745s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 17:27:07 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:53.477648699 +0000 UTC m=+133.035655806" watchObservedRunningTime="2026-07-21 17:27:53.478587848 +0000 UTC m=+133.036594957"
Jul 21 17:27:53.526263 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.526229    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:27:53.531356 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.531331    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.534169 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.534151    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 21 17:27:53.534356 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.534300    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 21 17:27:53.534469 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.534440    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 21 17:27:53.534628 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.534575    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 21 17:27:53.535314 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.535297    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 21 17:27:53.536295 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.535675    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 21 17:27:53.536865 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.536841    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 21 17:27:53.537005 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.536985    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 21 17:27:53.537173 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.537156    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 21 17:27:53.537284 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.537082    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ltcbj\""
Jul 21 17:27:53.537284 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.537260    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 21 17:27:53.537389 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.537044    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 21 17:27:53.537502 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.537465    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4fnb6prevqgk1\""
Jul 21 17:27:53.537834 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.537818    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 21 17:27:53.539714 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.539635    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-79f6958cdf-l4rs5" podStartSLOduration=2.502894747 podStartE2EDuration="6.539618425s" totalImagesPullingTime="4.036723678s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:53.536635242 +0000 UTC m=+133.094642350" watchObservedRunningTime="2026-07-21 17:27:53.539618425 +0000 UTC m=+133.097625532"
Jul 21 17:27:53.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.540104    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 21 17:27:53.541651 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.541632    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 21 17:27:53.553364 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.553335    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:27:53.586055 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586023    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586195 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586061    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586195 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586094    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586195 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586154    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586195 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586180    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586362 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586271    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586362 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586299    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586362 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586324    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586362 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586340    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586492 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586391    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586492 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-config\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586492 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586472    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7fhb\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-kube-api-access-g7fhb\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586598 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586513    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586598 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586598 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586598 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586568    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586598 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586583    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586780 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586780 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586631    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.586780 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.586659    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687382 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687568 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687568 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687568 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687568 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687568 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687568 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687894 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687593    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687894 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687616    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.687894 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.687647    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688240 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688210    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-config\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688383 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688260    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7fhb\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-kube-api-access-g7fhb\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688383 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688302    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688383 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688335    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688563 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688406    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688563 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688558    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688671 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688671 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688609    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688671 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.688853 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.688687    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.689283    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.689410    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.689745    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.690190    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.690689    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.691480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.692459    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.692466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.692494    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.692588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-web-config\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.692890 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.692665    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.693822 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.693766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.694051 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.694014    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.694143 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.694056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.694143 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.694086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-config\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.695650 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.695627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-config-out\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.696246 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.696221    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.696894 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.696465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.696894 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.696754    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.702183 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.702133    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7fhb\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-kube-api-access-g7fhb\") pod \"prometheus-k8s-0\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:53.844507 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:53.844293    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:27:54.045422 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:54.045286    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:27:54.052278 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:27:54.052235    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4b0d8ae_e125_4842_9881_8a6240f96766.slice/crio-2b131147ded45f98b155772057e0fc7ddd6b9075bbcb859afd99e559b798defc WatchSource:0}: Error finding container 2b131147ded45f98b155772057e0fc7ddd6b9075bbcb859afd99e559b798defc: Status 404 returned error can't find the container with id 2b131147ded45f98b155772057e0fc7ddd6b9075bbcb859afd99e559b798defc
Jul 21 17:27:54.470429 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:54.470393    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c" exitCode=0
Jul 21 17:27:54.470613 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:54.470480    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerDied","Data":"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"}
Jul 21 17:27:54.470613 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:54.470569    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerStarted","Data":"2b131147ded45f98b155772057e0fc7ddd6b9075bbcb859afd99e559b798defc"}
Jul 21 17:27:56.487861 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.487825    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerStarted","Data":"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9"}
Jul 21 17:27:56.488269 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.487963    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerStarted","Data":"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a"}
Jul 21 17:27:56.488269 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.488009    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerStarted","Data":"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8"}
Jul 21 17:27:56.488269 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.488050    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerStarted","Data":"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f"}
Jul 21 17:27:56.488269 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.488089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerStarted","Data":"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745"}
Jul 21 17:27:56.489911 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.489884    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd" event={"ID":"791f4d17-aa77-403b-a5a5-0314c63a72c7","Type":"ContainerStarted","Data":"a7ee77cab1f7cb4a3782894ff8e6ad18f97fab1db0f00635ae67c5cebd48b7e0"}
Jul 21 17:27:56.493389 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.493298    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rz8hm" event={"ID":"8cfe68ed-2ecc-4369-adcd-235f6b199b24","Type":"ContainerStarted","Data":"b8dab9f78cec9a29c989a063a2bb45e6a10ffd297fa92e3e4faa68c0c5be805a"}
Jul 21 17:27:56.493389 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.493343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-rz8hm" event={"ID":"8cfe68ed-2ecc-4369-adcd-235f6b199b24","Type":"ContainerStarted","Data":"7c3748398fff9409a7793c9f249668477a86490adcf8cb330f3571d78e61ad9a"}
Jul 21 17:27:56.497819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.497612    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl" event={"ID":"7b85463b-9b67-435a-96d7-d7f0f52f0782","Type":"ContainerStarted","Data":"f24b2808d779de9e9bcd06820e63fea8c2adbe733b212205907d3f6f43cef08d"}
Jul 21 17:27:56.497819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.497665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl" event={"ID":"7b85463b-9b67-435a-96d7-d7f0f52f0782","Type":"ContainerStarted","Data":"7f89bacb608cb684cbc9ee971f08827ba5a60136256865272f88567a76194a96"}
Jul 21 17:27:56.497819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.497693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl" event={"ID":"7b85463b-9b67-435a-96d7-d7f0f52f0782","Type":"ContainerStarted","Data":"a2d4252b17cd1fca0cb6c84c41e43fb6e47f78f305c30862c18a5023aea76329"}
Jul 21 17:27:56.510587 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.510477    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd" podStartSLOduration=3.392121452 podStartE2EDuration="5.510459782s" totalImagesPullingTime="2.11833833s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:56.509895412 +0000 UTC m=+136.067902522" watchObservedRunningTime="2026-07-21 17:27:56.510459782 +0000 UTC m=+136.068466892"
Jul 21 17:27:56.528210 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.528157    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-rz8hm" podStartSLOduration=134.371146869 podStartE2EDuration="2m16.528142477s" totalImagesPullingTime="2.156995608s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:25:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:56.527073475 +0000 UTC m=+136.085080591" watchObservedRunningTime="2026-07-21 17:27:56.528142477 +0000 UTC m=+136.086149660"
Jul 21 17:27:56.552145 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:56.552008    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7759f6578c-bnrrl" podStartSLOduration=1.6556945920000001 podStartE2EDuration="4.551994858s" totalImagesPullingTime="2.896300266s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:56.549717524 +0000 UTC m=+136.107724631" watchObservedRunningTime="2026-07-21 17:27:56.551994858 +0000 UTC m=+136.110001968"
Jul 21 17:27:57.508373 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:57.508323    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerStarted","Data":"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac"}
Jul 21 17:27:57.548344 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:57.548006    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.098846968 podStartE2EDuration="9.547983429s" totalImagesPullingTime="6.412223993s" totalInitContainerRuntime="36.912468ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:48 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:27:57.544262891 +0000 UTC m=+137.102269998" watchObservedRunningTime="2026-07-21 17:27:57.547983429 +0000 UTC m=+137.105990537"
Jul 21 17:27:59.521153 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:59.521123    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerStarted","Data":"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"}
Jul 21 17:27:59.521543 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:27:59.521190    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerStarted","Data":"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"}
Jul 21 17:28:00.529375 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:00.529339    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerStarted","Data":"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"}
Jul 21 17:28:00.529708 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:00.529463    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerStarted","Data":"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"}
Jul 21 17:28:01.536674 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:01.536641    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerStarted","Data":"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"}
Jul 21 17:28:01.537068 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:01.536701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerStarted","Data":"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"}
Jul 21 17:28:01.569457 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:01.569402    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.844876769 podStartE2EDuration="8.569386222s" totalImagesPullingTime="5.695553127s" totalInitContainerRuntime="28.956326ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:27:53 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:28:01.565003777 +0000 UTC m=+141.123010884" watchObservedRunningTime="2026-07-21 17:28:01.569386222 +0000 UTC m=+141.127393328"
Jul 21 17:28:03.108575 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.108515    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5574d67986-vk42j" podUID="b31c70f4-3128-4aa3-b468-29c378079b26" containerName="registry" containerID="cri-o://47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74" gracePeriod=30
Jul 21 17:28:03.389511 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.389480    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:28:03.487841 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.486889    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-trusted-ca\") pod \"b31c70f4-3128-4aa3-b468-29c378079b26\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") "
Jul 21 17:28:03.487841 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.486949    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-registry-certificates\") pod \"b31c70f4-3128-4aa3-b468-29c378079b26\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") "
Jul 21 17:28:03.487841 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.486988    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") pod \"b31c70f4-3128-4aa3-b468-29c378079b26\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") "
Jul 21 17:28:03.487841 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.487016    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h228x\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-kube-api-access-h228x\") pod \"b31c70f4-3128-4aa3-b468-29c378079b26\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") "
Jul 21 17:28:03.487841 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.487063    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-bound-sa-token\") pod \"b31c70f4-3128-4aa3-b468-29c378079b26\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") "
Jul 21 17:28:03.487841 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.487088    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31c70f4-3128-4aa3-b468-29c378079b26-ca-trust-extracted\") pod \"b31c70f4-3128-4aa3-b468-29c378079b26\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") "
Jul 21 17:28:03.487841 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.487118    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-installation-pull-secrets\") pod \"b31c70f4-3128-4aa3-b468-29c378079b26\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") "
Jul 21 17:28:03.487841 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.487183    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-image-registry-private-configuration\") pod \"b31c70f4-3128-4aa3-b468-29c378079b26\" (UID: \"b31c70f4-3128-4aa3-b468-29c378079b26\") "
Jul 21 17:28:03.488472 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.488436    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b31c70f4-3128-4aa3-b468-29c378079b26" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:03.488812 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.488759    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b31c70f4-3128-4aa3-b468-29c378079b26" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:28:03.490729 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.490681    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b31c70f4-3128-4aa3-b468-29c378079b26" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:03.490871 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.490849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b31c70f4-3128-4aa3-b468-29c378079b26" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:28:03.490937 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.490741    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-kube-api-access-h228x" (OuterVolumeSpecName: "kube-api-access-h228x") pod "b31c70f4-3128-4aa3-b468-29c378079b26" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26"). InnerVolumeSpecName "kube-api-access-h228x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:28:03.491219 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.491183    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b31c70f4-3128-4aa3-b468-29c378079b26" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:28:03.491402 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.491364    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b31c70f4-3128-4aa3-b468-29c378079b26" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:28:03.496192 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.496169    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b31c70f4-3128-4aa3-b468-29c378079b26-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b31c70f4-3128-4aa3-b468-29c378079b26" (UID: "b31c70f4-3128-4aa3-b468-29c378079b26"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:28:03.546217 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.546179    2563 generic.go:356] "Generic (PLEG): container finished" podID="b31c70f4-3128-4aa3-b468-29c378079b26" containerID="47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74" exitCode=0
Jul 21 17:28:03.546341 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.546245    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5574d67986-vk42j"
Jul 21 17:28:03.546341 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.546244    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5574d67986-vk42j" event={"ID":"b31c70f4-3128-4aa3-b468-29c378079b26","Type":"ContainerDied","Data":"47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74"}
Jul 21 17:28:03.546441 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.546384    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5574d67986-vk42j" event={"ID":"b31c70f4-3128-4aa3-b468-29c378079b26","Type":"ContainerDied","Data":"8b24894fd09e4df7caa310baead15f41edfb91bceb012af6b2167b2c0c44a068"}
Jul 21 17:28:03.546441 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.546410    2563 scope.go:117] "RemoveContainer" containerID="47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74"
Jul 21 17:28:03.556682 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.556662    2563 scope.go:117] "RemoveContainer" containerID="47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74"
Jul 21 17:28:03.556997 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:28:03.556977    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74\": container with ID starting with 47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74 not found: ID does not exist" containerID="47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74"
Jul 21 17:28:03.557082 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.557005    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74"} err="failed to get container status \"47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74\": rpc error: code = NotFound desc = could not find container \"47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74\": container with ID starting with 47dbf15536a340744e4b4afe490d40ad191201b6d36849ecd055d0e1f27a2b74 not found: ID does not exist"
Jul 21 17:28:03.571963 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.571931    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5574d67986-vk42j"]
Jul 21 17:28:03.575033 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.575010    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5574d67986-vk42j"]
Jul 21 17:28:03.588835 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.588809    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-bound-sa-token\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:28:03.588835 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.588831    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b31c70f4-3128-4aa3-b468-29c378079b26-ca-trust-extracted\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:28:03.588965 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.588841    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-installation-pull-secrets\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:28:03.588965 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.588851    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b31c70f4-3128-4aa3-b468-29c378079b26-image-registry-private-configuration\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:28:03.588965 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.588861    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-trusted-ca\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:28:03.588965 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.588869    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b31c70f4-3128-4aa3-b468-29c378079b26-registry-certificates\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:28:03.588965 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.588878    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-registry-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:28:03.588965 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.588887    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h228x\" (UniqueName: \"kubernetes.io/projected/b31c70f4-3128-4aa3-b468-29c378079b26-kube-api-access-h228x\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:28:03.844505 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:03.844476    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:04.911612 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:04.911577    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b31c70f4-3128-4aa3-b468-29c378079b26" path="/var/lib/kubelet/pods/b31c70f4-3128-4aa3-b468-29c378079b26/volumes"
Jul 21 17:28:12.128412 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:12.128376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:28:12.128840 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:12.128436    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:28:18.952041 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:28:18.951988    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qmpzt" podUID="e1a92e5b-3446-457c-bda1-e069f7fedc95"
Jul 21 17:28:19.611567 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:19.611540    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:28:23.853357 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:23.853303    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:28:23.855853 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:23.855829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/e1a92e5b-3446-457c-bda1-e069f7fedc95-metrics-tls\") pod \"dns-default-qmpzt\" (UID: \"e1a92e5b-3446-457c-bda1-e069f7fedc95\") " pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:28:24.115811 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:24.115710    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-c7lxh\""
Jul 21 17:28:24.123262 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:24.123245    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:28:24.285698 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:24.285673    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qmpzt"]
Jul 21 17:28:24.290054 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:28:24.290024    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode1a92e5b_3446_457c_bda1_e069f7fedc95.slice/crio-740c9f0e2212c1e6360d33e1b7c3b0a529abca22bc6fdce2b3906f39dd8c435d WatchSource:0}: Error finding container 740c9f0e2212c1e6360d33e1b7c3b0a529abca22bc6fdce2b3906f39dd8c435d: Status 404 returned error can't find the container with id 740c9f0e2212c1e6360d33e1b7c3b0a529abca22bc6fdce2b3906f39dd8c435d
Jul 21 17:28:24.632864 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:24.632828    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmpzt" event={"ID":"e1a92e5b-3446-457c-bda1-e069f7fedc95","Type":"ContainerStarted","Data":"740c9f0e2212c1e6360d33e1b7c3b0a529abca22bc6fdce2b3906f39dd8c435d"}
Jul 21 17:28:24.975510 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:24.971421    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32aeeb27-734e-431a-b23e-0b837e5e8e75/init-config-reloader/0.log"
Jul 21 17:28:25.171531 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:25.171499    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32aeeb27-734e-431a-b23e-0b837e5e8e75/alertmanager/0.log"
Jul 21 17:28:25.370885 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:25.370860    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32aeeb27-734e-431a-b23e-0b837e5e8e75/config-reloader/0.log"
Jul 21 17:28:25.571386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:25.571357    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32aeeb27-734e-431a-b23e-0b837e5e8e75/kube-rbac-proxy-web/0.log"
Jul 21 17:28:25.774023 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:25.773996    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32aeeb27-734e-431a-b23e-0b837e5e8e75/kube-rbac-proxy/0.log"
Jul 21 17:28:25.971093 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:25.971062    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32aeeb27-734e-431a-b23e-0b837e5e8e75/kube-rbac-proxy-metric/0.log"
Jul 21 17:28:26.171261 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.171238    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_32aeeb27-734e-431a-b23e-0b837e5e8e75/prom-label-proxy/0.log"
Jul 21 17:28:26.373190 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.373122    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-4fj4r_b8e304a3-94a7-4671-a188-fbc216c91b6f/cluster-monitoring-operator/0.log"
Jul 21 17:28:26.571196 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.571169    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvxb8_40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae/kube-state-metrics/0.log"
Jul 21 17:28:26.643677 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.643592    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmpzt" event={"ID":"e1a92e5b-3446-457c-bda1-e069f7fedc95","Type":"ContainerStarted","Data":"293eba48c87ca562829a1edba131141247d3e0e3944373e215fe415927470e12"}
Jul 21 17:28:26.643861 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.643679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qmpzt" event={"ID":"e1a92e5b-3446-457c-bda1-e069f7fedc95","Type":"ContainerStarted","Data":"bf9f5d3b4239a6081d5431f97e17626a48583e6c1eb160be5b36089256e6a3e5"}
Jul 21 17:28:26.643861 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.643711    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:28:26.662991 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.662941    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qmpzt" podStartSLOduration=129.972954177 podStartE2EDuration="2m11.662927934s" totalImagesPullingTime="1.689973757s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:26:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:28:26.661435796 +0000 UTC m=+166.219442903" watchObservedRunningTime="2026-07-21 17:28:26.662927934 +0000 UTC m=+166.220935422"
Jul 21 17:28:26.772310 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.772284    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvxb8_40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae/kube-rbac-proxy-main/0.log"
Jul 21 17:28:26.970158 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:26.970083    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvxb8_40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae/kube-rbac-proxy-self/0.log"
Jul 21 17:28:27.170606 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:27.170574    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bc5f9ff5d-8fnrd_791f4d17-aa77-403b-a5a5-0314c63a72c7/metrics-server/0.log"
Jul 21 17:28:27.571533 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:27.571503    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4z6f4_51b3d631-7caa-471a-a098-4fa381ed02d7/init-textfile/0.log"
Jul 21 17:28:27.771847 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:27.771820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4z6f4_51b3d631-7caa-471a-a098-4fa381ed02d7/node-exporter/0.log"
Jul 21 17:28:27.971197 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:27.971082    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4z6f4_51b3d631-7caa-471a-a098-4fa381ed02d7/kube-rbac-proxy/0.log"
Jul 21 17:28:29.371084 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:29.371038    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-l4rs5_dad95311-bc9c-41dc-88b7-637efe0ff6f9/kube-rbac-proxy-main/0.log"
Jul 21 17:28:29.571370 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:29.571347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-l4rs5_dad95311-bc9c-41dc-88b7-637efe0ff6f9/kube-rbac-proxy-self/0.log"
Jul 21 17:28:29.771273 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:29.771231    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-l4rs5_dad95311-bc9c-41dc-88b7-637efe0ff6f9/openshift-state-metrics/0.log"
Jul 21 17:28:29.970747 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:29.970721    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4b0d8ae-e125-4842-9881-8a6240f96766/init-config-reloader/0.log"
Jul 21 17:28:30.172401 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:30.172319    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4b0d8ae-e125-4842-9881-8a6240f96766/prometheus/0.log"
Jul 21 17:28:30.370889 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:30.370867    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4b0d8ae-e125-4842-9881-8a6240f96766/config-reloader/0.log"
Jul 21 17:28:30.570889 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:30.570864    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4b0d8ae-e125-4842-9881-8a6240f96766/thanos-sidecar/0.log"
Jul 21 17:28:30.771418 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:30.771393    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4b0d8ae-e125-4842-9881-8a6240f96766/kube-rbac-proxy-web/0.log"
Jul 21 17:28:30.970512 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:30.970443    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4b0d8ae-e125-4842-9881-8a6240f96766/kube-rbac-proxy/0.log"
Jul 21 17:28:31.170762 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:31.170697    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_e4b0d8ae-e125-4842-9881-8a6240f96766/kube-rbac-proxy-thanos/0.log"
Jul 21 17:28:31.372249 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:31.372219    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bhsbb_cdf7bfbf-2443-411f-af32-4c6f2ee61b65/prometheus-operator/0.log"
Jul 21 17:28:31.570958 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:31.570931    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bhsbb_cdf7bfbf-2443-411f-af32-4c6f2ee61b65/kube-rbac-proxy/0.log"
Jul 21 17:28:31.770559 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:31.770530    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-8wlkh_62bde93c-82cd-4ab9-8a93-b22780216f0f/prometheus-operator-admission-webhook/0.log"
Jul 21 17:28:31.970478 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:31.970430    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7759f6578c-bnrrl_7b85463b-9b67-435a-96d7-d7f0f52f0782/telemeter-client/0.log"
Jul 21 17:28:32.133978 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:32.133916    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:28:32.137949 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:32.137926    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7bc5f9ff5d-8fnrd"
Jul 21 17:28:32.172314 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:32.172290    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7759f6578c-bnrrl_7b85463b-9b67-435a-96d7-d7f0f52f0782/reload/0.log"
Jul 21 17:28:32.371255 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:32.371229    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7759f6578c-bnrrl_7b85463b-9b67-435a-96d7-d7f0f52f0782/kube-rbac-proxy/0.log"
Jul 21 17:28:33.770476 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:33.770450    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-dp25d_699ff256-14c0-4378-8721-9a322e2dc0ec/networking-console-plugin/0.log"
Jul 21 17:28:35.570395 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:35.570363    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmpzt_e1a92e5b-3446-457c-bda1-e069f7fedc95/dns/0.log"
Jul 21 17:28:35.770682 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:35.770655    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmpzt_e1a92e5b-3446-457c-bda1-e069f7fedc95/kube-rbac-proxy/0.log"
Jul 21 17:28:35.970554 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:35.970487    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5njds_27821ce6-26d8-4a08-b587-91eeac050017/dns-node-resolver/0.log"
Jul 21 17:28:36.650399 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:36.650372    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qmpzt"
Jul 21 17:28:53.844463 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:53.844426    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:53.861018 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:53.860993    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:28:54.777830 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:28:54.777766    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:07.573698 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.573658    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:07.574204 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.574097    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="alertmanager" containerID="cri-o://c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745" gracePeriod=120
Jul 21 17:29:07.574274 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.574227    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="config-reloader" containerID="cri-o://712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f" gracePeriod=120
Jul 21 17:29:07.574274 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.574241    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy" containerID="cri-o://fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a" gracePeriod=120
Jul 21 17:29:07.574372 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.574201    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy-web" containerID="cri-o://81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8" gracePeriod=120
Jul 21 17:29:07.574372 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.574232    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="prom-label-proxy" containerID="cri-o://3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac" gracePeriod=120
Jul 21 17:29:07.574460 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.574157    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy-metric" containerID="cri-o://510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9" gracePeriod=120
Jul 21 17:29:07.813644 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.813615    2563 generic.go:356] "Generic (PLEG): container finished" podID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerID="3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac" exitCode=0
Jul 21 17:29:07.813644 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.813640    2563 generic.go:356] "Generic (PLEG): container finished" podID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerID="fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a" exitCode=0
Jul 21 17:29:07.813843 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.813651    2563 generic.go:356] "Generic (PLEG): container finished" podID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerID="712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f" exitCode=0
Jul 21 17:29:07.813843 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.813660    2563 generic.go:356] "Generic (PLEG): container finished" podID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerID="c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745" exitCode=0
Jul 21 17:29:07.813843 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.813697    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerDied","Data":"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac"}
Jul 21 17:29:07.813843 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.813763    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerDied","Data":"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a"}
Jul 21 17:29:07.813843 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.813817    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerDied","Data":"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f"}
Jul 21 17:29:07.813995 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:07.813848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerDied","Data":"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745"}
Jul 21 17:29:08.820039 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.820013    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:08.820352 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.820300    2563 generic.go:356] "Generic (PLEG): container finished" podID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerID="510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9" exitCode=0
Jul 21 17:29:08.820352 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.820317    2563 generic.go:356] "Generic (PLEG): container finished" podID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerID="81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8" exitCode=0
Jul 21 17:29:08.820424 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.820360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerDied","Data":"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9"}
Jul 21 17:29:08.820457 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.820435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerDied","Data":"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8"}
Jul 21 17:29:08.820489 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.820466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"32aeeb27-734e-431a-b23e-0b837e5e8e75","Type":"ContainerDied","Data":"1e560aeec0fde756d5f5a8bd0f1052ebe9c0e7dbd54f9ffb4ee1acd5767a9172"}
Jul 21 17:29:08.820525 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.820500    2563 scope.go:117] "RemoveContainer" containerID="3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac"
Jul 21 17:29:08.828510 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.828492    2563 scope.go:117] "RemoveContainer" containerID="510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9"
Jul 21 17:29:08.835680 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.835663    2563 scope.go:117] "RemoveContainer" containerID="fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a"
Jul 21 17:29:08.842244 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.842229    2563 scope.go:117] "RemoveContainer" containerID="81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8"
Jul 21 17:29:08.849129 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.849102    2563 scope.go:117] "RemoveContainer" containerID="712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f"
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.856715    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.856777    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-metrics-client-ca\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857073    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-web-config\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857117    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-metric\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857164    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857224    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-main-db\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857254    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-out\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857292    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-cluster-tls-config\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-web\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857347    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5j8h\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-kube-api-access-f5j8h\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857407    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-tls-assets\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857439    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-volume\") pod \"32aeeb27-734e-431a-b23e-0b837e5e8e75\" (UID: \"32aeeb27-734e-431a-b23e-0b837e5e8e75\") "
Jul 21 17:29:08.860205 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.857955    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:08.862806 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.860932    2563 scope.go:117] "RemoveContainer" containerID="c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745"
Jul 21 17:29:08.862806 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.861556    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:29:08.862806 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.861937    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:08.864366 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.863907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-volume" (OuterVolumeSpecName: "config-volume") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:08.866046 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.866007    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-out" (OuterVolumeSpecName: "config-out") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:29:08.866136 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.866048    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:08.866136 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.866064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:08.866136 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.866102    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:08.868547 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.868512    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:08.870281 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.870014    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:08.872193 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.872155    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:29:08.872899 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.872865    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-kube-api-access-f5j8h" (OuterVolumeSpecName: "kube-api-access-f5j8h") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "kube-api-access-f5j8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:29:08.878684 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.878659    2563 scope.go:117] "RemoveContainer" containerID="28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517"
Jul 21 17:29:08.879984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.879967    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-web-config" (OuterVolumeSpecName: "web-config") pod "32aeeb27-734e-431a-b23e-0b837e5e8e75" (UID: "32aeeb27-734e-431a-b23e-0b837e5e8e75"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:08.886591 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.886575    2563 scope.go:117] "RemoveContainer" containerID="3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac"
Jul 21 17:29:08.886872 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:08.886851    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac\": container with ID starting with 3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac not found: ID does not exist" containerID="3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac"
Jul 21 17:29:08.886938 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.886887    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac"} err="failed to get container status \"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac\": rpc error: code = NotFound desc = could not find container \"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac\": container with ID starting with 3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac not found: ID does not exist"
Jul 21 17:29:08.886938 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.886918    2563 scope.go:117] "RemoveContainer" containerID="510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9"
Jul 21 17:29:08.887188 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:08.887139    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9\": container with ID starting with 510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9 not found: ID does not exist" containerID="510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9"
Jul 21 17:29:08.887275 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.887190    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9"} err="failed to get container status \"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9\": rpc error: code = NotFound desc = could not find container \"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9\": container with ID starting with 510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9 not found: ID does not exist"
Jul 21 17:29:08.887275 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.887230    2563 scope.go:117] "RemoveContainer" containerID="fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a"
Jul 21 17:29:08.887549 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:08.887529    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a\": container with ID starting with fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a not found: ID does not exist" containerID="fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a"
Jul 21 17:29:08.887629 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.887581    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a"} err="failed to get container status \"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a\": rpc error: code = NotFound desc = could not find container \"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a\": container with ID starting with fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a not found: ID does not exist"
Jul 21 17:29:08.887629 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.887599    2563 scope.go:117] "RemoveContainer" containerID="81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8"
Jul 21 17:29:08.887960 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:08.887937    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8\": container with ID starting with 81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8 not found: ID does not exist" containerID="81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8"
Jul 21 17:29:08.888039 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.887967    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8"} err="failed to get container status \"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8\": rpc error: code = NotFound desc = could not find container \"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8\": container with ID starting with 81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8 not found: ID does not exist"
Jul 21 17:29:08.888039 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.887985    2563 scope.go:117] "RemoveContainer" containerID="712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f"
Jul 21 17:29:08.888254 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:08.888237    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f\": container with ID starting with 712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f not found: ID does not exist" containerID="712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f"
Jul 21 17:29:08.888304 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.888257    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f"} err="failed to get container status \"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f\": rpc error: code = NotFound desc = could not find container \"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f\": container with ID starting with 712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f not found: ID does not exist"
Jul 21 17:29:08.888304 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.888270    2563 scope.go:117] "RemoveContainer" containerID="c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745"
Jul 21 17:29:08.888550 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:08.888529    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745\": container with ID starting with c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745 not found: ID does not exist" containerID="c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745"
Jul 21 17:29:08.888610 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.888558    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745"} err="failed to get container status \"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745\": rpc error: code = NotFound desc = could not find container \"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745\": container with ID starting with c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745 not found: ID does not exist"
Jul 21 17:29:08.888610 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.888577    2563 scope.go:117] "RemoveContainer" containerID="28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517"
Jul 21 17:29:08.888845 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:08.888820    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517\": container with ID starting with 28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517 not found: ID does not exist" containerID="28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517"
Jul 21 17:29:08.888912 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.888852    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517"} err="failed to get container status \"28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517\": rpc error: code = NotFound desc = could not find container \"28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517\": container with ID starting with 28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517 not found: ID does not exist"
Jul 21 17:29:08.888912 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.888874    2563 scope.go:117] "RemoveContainer" containerID="3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac"
Jul 21 17:29:08.889156 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.889137    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac"} err="failed to get container status \"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac\": rpc error: code = NotFound desc = could not find container \"3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac\": container with ID starting with 3b73170b43ccf9c42b4ee9dcaceaabc6d83e4b1484be02ba2d31ae0cbf73c1ac not found: ID does not exist"
Jul 21 17:29:08.889218 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.889157    2563 scope.go:117] "RemoveContainer" containerID="510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9"
Jul 21 17:29:08.889417 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.889392    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9"} err="failed to get container status \"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9\": rpc error: code = NotFound desc = could not find container \"510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9\": container with ID starting with 510e9d6bbaa79fadf6a5872c5276a8e28b4a92badfe11e9b72aafa5b0a8938a9 not found: ID does not exist"
Jul 21 17:29:08.889456 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.889420    2563 scope.go:117] "RemoveContainer" containerID="fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a"
Jul 21 17:29:08.889651 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.889632    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a"} err="failed to get container status \"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a\": rpc error: code = NotFound desc = could not find container \"fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a\": container with ID starting with fcef86381fc1ad5b56062874f0d65adef94e179ad1da861385f9a40f9ee3364a not found: ID does not exist"
Jul 21 17:29:08.889728 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.889653    2563 scope.go:117] "RemoveContainer" containerID="81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8"
Jul 21 17:29:08.889908 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.889889    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8"} err="failed to get container status \"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8\": rpc error: code = NotFound desc = could not find container \"81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8\": container with ID starting with 81f4d0f382288bd991035ef7cb3793d5665418dc968e95025c024e59a3b227a8 not found: ID does not exist"
Jul 21 17:29:08.889965 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.889908    2563 scope.go:117] "RemoveContainer" containerID="712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f"
Jul 21 17:29:08.890122 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.890102    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f"} err="failed to get container status \"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f\": rpc error: code = NotFound desc = could not find container \"712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f\": container with ID starting with 712fc22a1bcc08ece52954982a162a688f4e80a9968dd7c5e49c4c92c048a20f not found: ID does not exist"
Jul 21 17:29:08.890161 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.890124    2563 scope.go:117] "RemoveContainer" containerID="c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745"
Jul 21 17:29:08.890342 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.890324    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745"} err="failed to get container status \"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745\": rpc error: code = NotFound desc = could not find container \"c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745\": container with ID starting with c7db4c73fca0b680de08e5a7f3579dd01f249ac6edebf02b7471d67d3f6b8745 not found: ID does not exist"
Jul 21 17:29:08.890342 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.890342    2563 scope.go:117] "RemoveContainer" containerID="28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517"
Jul 21 17:29:08.890590 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.890572    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517"} err="failed to get container status \"28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517\": rpc error: code = NotFound desc = could not find container \"28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517\": container with ID starting with 28913984854f00c164800e76714e5c7d36199debe5ba6c6c9ba16c9ff7a17517 not found: ID does not exist"
Jul 21 17:29:08.958671 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958640    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-main-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958671 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958667    2563 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-metrics-client-ca\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958680    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-web-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958693    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958705    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958718    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958730    2563 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-alertmanager-main-db\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958741    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-out\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958754    2563 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-cluster-tls-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958766    2563 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958779    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5j8h\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-kube-api-access-f5j8h\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958817    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/32aeeb27-734e-431a-b23e-0b837e5e8e75-tls-assets\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:08.958875 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:08.958829    2563 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/32aeeb27-734e-431a-b23e-0b837e5e8e75-config-volume\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:09.825434 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.825409    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.849348 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.849316    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:09.856073 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.856049    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:09.884206 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884179    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:09.884466 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884453    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="alertmanager"
Jul 21 17:29:09.884466 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884467    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="alertmanager"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884476    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="prom-label-proxy"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884482    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="prom-label-proxy"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884494    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884499    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884507    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy-metric"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884512    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy-metric"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884518    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b31c70f4-3128-4aa3-b468-29c378079b26" containerName="registry"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884522    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b31c70f4-3128-4aa3-b468-29c378079b26" containerName="registry"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884532    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy-web"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884537    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy-web"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884545    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="init-config-reloader"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884550    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="init-config-reloader"
Jul 21 17:29:09.884555 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884561    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="config-reloader"
Jul 21 17:29:09.884989 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884568    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="config-reloader"
Jul 21 17:29:09.884989 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884609    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy"
Jul 21 17:29:09.884989 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884616    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy-metric"
Jul 21 17:29:09.884989 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884621    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b31c70f4-3128-4aa3-b468-29c378079b26" containerName="registry"
Jul 21 17:29:09.884989 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884627    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="alertmanager"
Jul 21 17:29:09.884989 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884632    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="prom-label-proxy"
Jul 21 17:29:09.884989 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884638    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="config-reloader"
Jul 21 17:29:09.884989 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.884643    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" containerName="kube-rbac-proxy-web"
Jul 21 17:29:09.890184 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.890166    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.893093 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.893030    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-z9cgb\""
Jul 21 17:29:09.893093 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.893055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 21 17:29:09.893093 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.893055    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 21 17:29:09.893303 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.893102    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 21 17:29:09.893303 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.893298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 21 17:29:09.893780 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.893766    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 21 17:29:09.894005 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.893989    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 21 17:29:09.894056 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.893991    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 21 17:29:09.895474 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.895441    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 21 17:29:09.899606 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.899590    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 21 17:29:09.907488 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.907467    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:09.968658 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968813 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968661    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968813 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968813 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968813 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968759    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968817    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-config-out\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968859    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968901    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968923    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqp7m\" (UniqueName: \"kubernetes.io/projected/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-kube-api-access-bqp7m\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.968947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.969099 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.968989    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-web-config\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.969099 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.969006    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:09.969099 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:09.969032    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070048 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070048 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070051    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070224 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070077    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqp7m\" (UniqueName: \"kubernetes.io/projected/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-kube-api-access-bqp7m\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070224 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070224 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070137    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-web-config\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070224 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070397 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070397 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070293    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070397 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070397 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070585 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070397    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070585 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070430    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070585 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070464    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-config-out\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.070734 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.070638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.071037 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.071006    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.073277 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.073251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-config-out\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.073381 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.073357    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.073450 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.073402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.073701 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.073648    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.073812 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.073768    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.073882 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.073869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-config-volume\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.073970 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.073946    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.074204 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.074186    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-web-config\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.074348 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.074326    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.075434 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.075411    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.078650 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.078630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqp7m\" (UniqueName: \"kubernetes.io/projected/ff93ea2e-915b-4ca9-9605-8e3c9eff5c16-kube-api-access-bqp7m\") pod \"alertmanager-main-0\" (UID: \"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.199601 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.199541    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 21 17:29:10.342074 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.342043    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 21 17:29:10.346242 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:29:10.346218    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff93ea2e_915b_4ca9_9605_8e3c9eff5c16.slice/crio-dee2ff6f28dedcca4d256358a0b3885d458086dad4541715753af38f904649dc WatchSource:0}: Error finding container dee2ff6f28dedcca4d256358a0b3885d458086dad4541715753af38f904649dc: Status 404 returned error can't find the container with id dee2ff6f28dedcca4d256358a0b3885d458086dad4541715753af38f904649dc
Jul 21 17:29:10.833569 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.833536    2563 generic.go:356] "Generic (PLEG): container finished" podID="ff93ea2e-915b-4ca9-9605-8e3c9eff5c16" containerID="f3c0660eb550a939ba992df9917067211af2b7a08061d2bd304e38199c30ab97" exitCode=0
Jul 21 17:29:10.833969 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.833601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16","Type":"ContainerDied","Data":"f3c0660eb550a939ba992df9917067211af2b7a08061d2bd304e38199c30ab97"}
Jul 21 17:29:10.833969 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.833680    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16","Type":"ContainerStarted","Data":"dee2ff6f28dedcca4d256358a0b3885d458086dad4541715753af38f904649dc"}
Jul 21 17:29:10.909447 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:10.909419    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32aeeb27-734e-431a-b23e-0b837e5e8e75" path="/var/lib/kubelet/pods/32aeeb27-734e-431a-b23e-0b837e5e8e75/volumes"
Jul 21 17:29:11.822663 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.822626    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:29:11.823047 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.823012    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="prometheus" containerID="cri-o://91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f" gracePeriod=600
Jul 21 17:29:11.823172 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.823037    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="config-reloader" containerID="cri-o://8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203" gracePeriod=600
Jul 21 17:29:11.823172 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.823073    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy-web" containerID="cri-o://f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf" gracePeriod=600
Jul 21 17:29:11.823172 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.823034    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy" containerID="cri-o://99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a" gracePeriod=600
Jul 21 17:29:11.823318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.823043    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="thanos-sidecar" containerID="cri-o://090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c" gracePeriod=600
Jul 21 17:29:11.823318 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.823086    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy-thanos" containerID="cri-o://2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65" gracePeriod=600
Jul 21 17:29:11.846832 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.846633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16","Type":"ContainerStarted","Data":"6a4ca9f656d0af84953680b9495bc3b21f5e35a944b824624764c1d1c51b016b"}
Jul 21 17:29:11.846832 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.846714    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16","Type":"ContainerStarted","Data":"7e0a43a50c8001fdb702eec829fc737960eea3e10245b4dbca1f717b94179ca0"}
Jul 21 17:29:11.846832 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.846757    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16","Type":"ContainerStarted","Data":"a1c912f5ca0eb73920231db2217589ca26c92e118cd4ffb27055dfeb906823e8"}
Jul 21 17:29:11.846832 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.846809    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16","Type":"ContainerStarted","Data":"4544aaecb69f9d12307f10bed4a7b723a98444143353f727e3d6ccf44c983fb8"}
Jul 21 17:29:11.847373 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.846842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16","Type":"ContainerStarted","Data":"11ecc1d688c31894442c9cd00c60469ead62e3114e77fa75e08d7629bb98fb0a"}
Jul 21 17:29:11.847373 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.846871    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ff93ea2e-915b-4ca9-9605-8e3c9eff5c16","Type":"ContainerStarted","Data":"1defd97a998e7e6d2369e0bdefb5ce9420dd6af9f7d9aea3212b59a82d1d1eb5"}
Jul 21 17:29:11.882170 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:11.882126    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.86867926 podStartE2EDuration="2.882109252s" totalImagesPullingTime="0s" totalInitContainerRuntime="13.429992ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:29:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:29:11.881337387 +0000 UTC m=+211.439344485" watchObservedRunningTime="2026-07-21 17:29:11.882109252 +0000 UTC m=+211.440116359"
Jul 21 17:29:12.091409 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.091381    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.189316 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189284    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-config-out\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189330    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-0\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189348    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-2\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-db\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189386    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-metrics-client-ca\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g7fhb\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-kube-api-access-g7fhb\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189430    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-kube-rbac-proxy\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189490 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189458    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-thanos-prometheus-http-client-file\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189880 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189498    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189880 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189533    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-trusted-ca-bundle\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189880 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189571    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-serving-certs-ca-bundle\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189880 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189599    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-1\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189880 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189623    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-tls-assets\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.189880 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189701    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:12.189880 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.189854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:12.190242 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.190048    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:12.190310 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.190279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:12.190454 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.190433    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191455    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191510    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-kubelet-serving-ca-bundle\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191561    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-grpc-tls\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191563    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191602    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-config\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191661    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-tls\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-web-config\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191859    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-metrics-client-certs\") pod \"e4b0d8ae-e125-4842-9881-8a6240f96766\" (UID: \"e4b0d8ae-e125-4842-9881-8a6240f96766\") "
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.191923    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192546    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192568    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192582    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192596    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-db\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192608    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-metrics-client-ca\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192622    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192634    2563 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.192904 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192647    2563 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/e4b0d8ae-e125-4842-9881-8a6240f96766-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.194313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192850    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-config-out" (OuterVolumeSpecName: "config-out") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:29:12.194313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.192957    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:29:12.194313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.193292    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-kube-api-access-g7fhb" (OuterVolumeSpecName: "kube-api-access-g7fhb") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "kube-api-access-g7fhb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:29:12.194313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.193373    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.194313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.193401    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.194313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.194209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.194618 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.194419    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.194685 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.194648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.195113 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.195088    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-config" (OuterVolumeSpecName: "config") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.195864 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.195764    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.195864 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.195779    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.205775 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.205753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-web-config" (OuterVolumeSpecName: "web-config") pod "e4b0d8ae-e125-4842-9881-8a6240f96766" (UID: "e4b0d8ae-e125-4842-9881-8a6240f96766"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:29:12.293356 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293324    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293356 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293350    2563 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-tls-assets\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293356 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293363    2563 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-grpc-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293374    2563 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293386    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293399    2563 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293411    2563 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-web-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293423    2563 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-metrics-client-certs\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293433    2563 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/e4b0d8ae-e125-4842-9881-8a6240f96766-config-out\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293444    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7fhb\" (UniqueName: \"kubernetes.io/projected/e4b0d8ae-e125-4842-9881-8a6240f96766-kube-api-access-g7fhb\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293459    2563 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-secret-kube-rbac-proxy\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.293572 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.293472    2563 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/e4b0d8ae-e125-4842-9881-8a6240f96766-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:29:12.853323 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853291    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65" exitCode=0
Jul 21 17:29:12.853323 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853315    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a" exitCode=0
Jul 21 17:29:12.853323 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853322    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf" exitCode=0
Jul 21 17:29:12.853323 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853329    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c" exitCode=0
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853336    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203" exitCode=0
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853341    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f" exitCode=0
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853376    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853376    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerDied","Data":"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"}
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853496    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerDied","Data":"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"}
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerDied","Data":"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"}
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853558    2563 scope.go:117] "RemoveContainer" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853583    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerDied","Data":"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"}
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853623    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerDied","Data":"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"}
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853663    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerDied","Data":"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"}
Jul 21 17:29:12.853819 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.853703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"e4b0d8ae-e125-4842-9881-8a6240f96766","Type":"ContainerDied","Data":"2b131147ded45f98b155772057e0fc7ddd6b9075bbcb859afd99e559b798defc"}
Jul 21 17:29:12.861901 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.861878    2563 scope.go:117] "RemoveContainer" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"
Jul 21 17:29:12.868978 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.868961    2563 scope.go:117] "RemoveContainer" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"
Jul 21 17:29:12.875748 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.875733    2563 scope.go:117] "RemoveContainer" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"
Jul 21 17:29:12.879037 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.879015    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:29:12.883349 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.883330    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:29:12.883584 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.883562    2563 scope.go:117] "RemoveContainer" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"
Jul 21 17:29:12.890466 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.890447    2563 scope.go:117] "RemoveContainer" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"
Jul 21 17:29:12.897566 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.897545    2563 scope.go:117] "RemoveContainer" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"
Jul 21 17:29:12.904477 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.904462    2563 scope.go:117] "RemoveContainer" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"
Jul 21 17:29:12.904720 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:12.904700    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": container with ID starting with 2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65 not found: ID does not exist" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"
Jul 21 17:29:12.904781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.904729    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"} err="failed to get container status \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": rpc error: code = NotFound desc = could not find container \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": container with ID starting with 2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65 not found: ID does not exist"
Jul 21 17:29:12.904781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.904746    2563 scope.go:117] "RemoveContainer" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"
Jul 21 17:29:12.904979 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:12.904964    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": container with ID starting with 99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a not found: ID does not exist" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"
Jul 21 17:29:12.905019 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.904983    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"} err="failed to get container status \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": rpc error: code = NotFound desc = could not find container \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": container with ID starting with 99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a not found: ID does not exist"
Jul 21 17:29:12.905019 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.904996    2563 scope.go:117] "RemoveContainer" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"
Jul 21 17:29:12.905268 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:12.905250    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": container with ID starting with f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf not found: ID does not exist" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"
Jul 21 17:29:12.905333 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.905271    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"} err="failed to get container status \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": rpc error: code = NotFound desc = could not find container \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": container with ID starting with f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf not found: ID does not exist"
Jul 21 17:29:12.905333 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.905284    2563 scope.go:117] "RemoveContainer" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"
Jul 21 17:29:12.905525 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:12.905500    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": container with ID starting with 090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c not found: ID does not exist" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"
Jul 21 17:29:12.905577 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.905527    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"} err="failed to get container status \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": rpc error: code = NotFound desc = could not find container \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": container with ID starting with 090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c not found: ID does not exist"
Jul 21 17:29:12.905577 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.905538    2563 scope.go:117] "RemoveContainer" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"
Jul 21 17:29:12.905763 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:12.905744    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": container with ID starting with 8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203 not found: ID does not exist" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"
Jul 21 17:29:12.905891 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.905766    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"} err="failed to get container status \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": rpc error: code = NotFound desc = could not find container \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": container with ID starting with 8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203 not found: ID does not exist"
Jul 21 17:29:12.905942 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.905926    2563 scope.go:117] "RemoveContainer" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"
Jul 21 17:29:12.906208 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:12.906192    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": container with ID starting with 91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f not found: ID does not exist" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"
Jul 21 17:29:12.906262 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.906212    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"} err="failed to get container status \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": rpc error: code = NotFound desc = could not find container \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": container with ID starting with 91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f not found: ID does not exist"
Jul 21 17:29:12.906262 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.906231    2563 scope.go:117] "RemoveContainer" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"
Jul 21 17:29:12.906472 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:29:12.906456    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": container with ID starting with 3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c not found: ID does not exist" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"
Jul 21 17:29:12.906526 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.906474    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"} err="failed to get container status \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": rpc error: code = NotFound desc = could not find container \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": container with ID starting with 3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c not found: ID does not exist"
Jul 21 17:29:12.906526 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.906490    2563 scope.go:117] "RemoveContainer" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"
Jul 21 17:29:12.906714 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.906700    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"} err="failed to get container status \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": rpc error: code = NotFound desc = could not find container \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": container with ID starting with 2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65 not found: ID does not exist"
Jul 21 17:29:12.906714 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.906714    2563 scope.go:117] "RemoveContainer" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"
Jul 21 17:29:12.906992 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.906971    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"} err="failed to get container status \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": rpc error: code = NotFound desc = could not find container \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": container with ID starting with 99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a not found: ID does not exist"
Jul 21 17:29:12.907105 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.906992    2563 scope.go:117] "RemoveContainer" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"
Jul 21 17:29:12.907224 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.907207    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"} err="failed to get container status \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": rpc error: code = NotFound desc = could not find container \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": container with ID starting with f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf not found: ID does not exist"
Jul 21 17:29:12.907267 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.907225    2563 scope.go:117] "RemoveContainer" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"
Jul 21 17:29:12.907493 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.907441    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"} err="failed to get container status \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": rpc error: code = NotFound desc = could not find container \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": container with ID starting with 090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c not found: ID does not exist"
Jul 21 17:29:12.907493 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.907460    2563 scope.go:117] "RemoveContainer" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"
Jul 21 17:29:12.908772 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.908032    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"} err="failed to get container status \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": rpc error: code = NotFound desc = could not find container \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": container with ID starting with 8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203 not found: ID does not exist"
Jul 21 17:29:12.908772 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.908061    2563 scope.go:117] "RemoveContainer" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"
Jul 21 17:29:12.908772 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.908293    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"} err="failed to get container status \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": rpc error: code = NotFound desc = could not find container \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": container with ID starting with 91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f not found: ID does not exist"
Jul 21 17:29:12.908772 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.908312    2563 scope.go:117] "RemoveContainer" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"
Jul 21 17:29:12.908772 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.908682    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"} err="failed to get container status \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": rpc error: code = NotFound desc = could not find container \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": container with ID starting with 3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c not found: ID does not exist"
Jul 21 17:29:12.908772 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.908734    2563 scope.go:117] "RemoveContainer" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"
Jul 21 17:29:12.909353 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.909085    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" path="/var/lib/kubelet/pods/e4b0d8ae-e125-4842-9881-8a6240f96766/volumes"
Jul 21 17:29:12.909353 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.909110    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"} err="failed to get container status \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": rpc error: code = NotFound desc = could not find container \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": container with ID starting with 2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65 not found: ID does not exist"
Jul 21 17:29:12.909353 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.909144    2563 scope.go:117] "RemoveContainer" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"
Jul 21 17:29:12.909610 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.909590    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"} err="failed to get container status \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": rpc error: code = NotFound desc = could not find container \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": container with ID starting with 99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a not found: ID does not exist"
Jul 21 17:29:12.909668 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.909610    2563 scope.go:117] "RemoveContainer" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"
Jul 21 17:29:12.909905 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.909883    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"} err="failed to get container status \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": rpc error: code = NotFound desc = could not find container \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": container with ID starting with f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf not found: ID does not exist"
Jul 21 17:29:12.909905 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.909904    2563 scope.go:117] "RemoveContainer" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"
Jul 21 17:29:12.910081 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910054    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:29:12.910152 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910131    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"} err="failed to get container status \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": rpc error: code = NotFound desc = could not find container \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": container with ID starting with 090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c not found: ID does not exist"
Jul 21 17:29:12.910188 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910156    2563 scope.go:117] "RemoveContainer" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"
Jul 21 17:29:12.910371 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910357    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="thanos-sidecar"
Jul 21 17:29:12.910371 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910372    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="thanos-sidecar"
Jul 21 17:29:12.910371 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910380    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy-thanos"
Jul 21 17:29:12.910482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910386    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy-thanos"
Jul 21 17:29:12.910482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910408    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy"
Jul 21 17:29:12.910482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910416    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy"
Jul 21 17:29:12.910482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910427    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="init-config-reloader"
Jul 21 17:29:12.910482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910434    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="init-config-reloader"
Jul 21 17:29:12.910482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910451    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="config-reloader"
Jul 21 17:29:12.910482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910429    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"} err="failed to get container status \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": rpc error: code = NotFound desc = could not find container \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": container with ID starting with 8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203 not found: ID does not exist"
Jul 21 17:29:12.910482 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910472    2563 scope.go:117] "RemoveContainer" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910461    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="config-reloader"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910531    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="prometheus"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910542    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="prometheus"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910550    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy-web"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910556    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy-web"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910639    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy-web"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910652    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="thanos-sidecar"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910660    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy-thanos"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910667    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="kube-rbac-proxy"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910673    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="prometheus"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910677    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4b0d8ae-e125-4842-9881-8a6240f96766" containerName="config-reloader"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910726    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"} err="failed to get container status \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": rpc error: code = NotFound desc = could not find container \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": container with ID starting with 91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f not found: ID does not exist"
Jul 21 17:29:12.910755 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910742    2563 scope.go:117] "RemoveContainer" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"
Jul 21 17:29:12.911236 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.910987    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"} err="failed to get container status \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": rpc error: code = NotFound desc = could not find container \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": container with ID starting with 3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c not found: ID does not exist"
Jul 21 17:29:12.911236 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.911002    2563 scope.go:117] "RemoveContainer" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"
Jul 21 17:29:12.911236 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.911196    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"} err="failed to get container status \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": rpc error: code = NotFound desc = could not find container \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": container with ID starting with 2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65 not found: ID does not exist"
Jul 21 17:29:12.911236 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.911217    2563 scope.go:117] "RemoveContainer" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"
Jul 21 17:29:12.911444 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.911425    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"} err="failed to get container status \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": rpc error: code = NotFound desc = could not find container \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": container with ID starting with 99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a not found: ID does not exist"
Jul 21 17:29:12.911483 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.911446    2563 scope.go:117] "RemoveContainer" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"
Jul 21 17:29:12.911700 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.911684    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"} err="failed to get container status \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": rpc error: code = NotFound desc = could not find container \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": container with ID starting with f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf not found: ID does not exist"
Jul 21 17:29:12.911751 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.911699    2563 scope.go:117] "RemoveContainer" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"
Jul 21 17:29:12.912169 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.912145    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"} err="failed to get container status \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": rpc error: code = NotFound desc = could not find container \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": container with ID starting with 090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c not found: ID does not exist"
Jul 21 17:29:12.912291 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.912170    2563 scope.go:117] "RemoveContainer" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"
Jul 21 17:29:12.912448 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.912430    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"} err="failed to get container status \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": rpc error: code = NotFound desc = could not find container \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": container with ID starting with 8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203 not found: ID does not exist"
Jul 21 17:29:12.912515 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.912448    2563 scope.go:117] "RemoveContainer" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"
Jul 21 17:29:12.912660 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.912642    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"} err="failed to get container status \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": rpc error: code = NotFound desc = could not find container \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": container with ID starting with 91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f not found: ID does not exist"
Jul 21 17:29:12.912735 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.912663    2563 scope.go:117] "RemoveContainer" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"
Jul 21 17:29:12.912928 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.912910    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"} err="failed to get container status \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": rpc error: code = NotFound desc = could not find container \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": container with ID starting with 3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c not found: ID does not exist"
Jul 21 17:29:12.912974 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.912929    2563 scope.go:117] "RemoveContainer" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"
Jul 21 17:29:12.913142 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.913126    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"} err="failed to get container status \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": rpc error: code = NotFound desc = could not find container \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": container with ID starting with 2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65 not found: ID does not exist"
Jul 21 17:29:12.913192 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.913143    2563 scope.go:117] "RemoveContainer" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"
Jul 21 17:29:12.913558 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.913537    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"} err="failed to get container status \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": rpc error: code = NotFound desc = could not find container \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": container with ID starting with 99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a not found: ID does not exist"
Jul 21 17:29:12.913644 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.913559    2563 scope.go:117] "RemoveContainer" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"
Jul 21 17:29:12.913852 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.913831    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"} err="failed to get container status \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": rpc error: code = NotFound desc = could not find container \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": container with ID starting with f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf not found: ID does not exist"
Jul 21 17:29:12.913902 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.913853    2563 scope.go:117] "RemoveContainer" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"
Jul 21 17:29:12.914059 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.914042    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"} err="failed to get container status \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": rpc error: code = NotFound desc = could not find container \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": container with ID starting with 090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c not found: ID does not exist"
Jul 21 17:29:12.914098 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.914061    2563 scope.go:117] "RemoveContainer" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"
Jul 21 17:29:12.914298 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.914281    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"} err="failed to get container status \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": rpc error: code = NotFound desc = could not find container \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": container with ID starting with 8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203 not found: ID does not exist"
Jul 21 17:29:12.914369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.914300    2563 scope.go:117] "RemoveContainer" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"
Jul 21 17:29:12.914512 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.914495    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"} err="failed to get container status \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": rpc error: code = NotFound desc = could not find container \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": container with ID starting with 91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f not found: ID does not exist"
Jul 21 17:29:12.914512 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.914512    2563 scope.go:117] "RemoveContainer" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"
Jul 21 17:29:12.914745 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.914724    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"} err="failed to get container status \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": rpc error: code = NotFound desc = could not find container \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": container with ID starting with 3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c not found: ID does not exist"
Jul 21 17:29:12.914745 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.914744    2563 scope.go:117] "RemoveContainer" containerID="2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"
Jul 21 17:29:12.915032 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.915004    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65"} err="failed to get container status \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": rpc error: code = NotFound desc = could not find container \"2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65\": container with ID starting with 2a162dc8a61d96f6d9c1ecba7140c8c21ec9f1ecad5823219097aae37def6a65 not found: ID does not exist"
Jul 21 17:29:12.915032 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.915031    2563 scope.go:117] "RemoveContainer" containerID="99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"
Jul 21 17:29:12.915285 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.915269    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a"} err="failed to get container status \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": rpc error: code = NotFound desc = could not find container \"99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a\": container with ID starting with 99ca5eeea7681495c65f0196ef2bed25f4eabd50d03dfc340d21c1f407d1593a not found: ID does not exist"
Jul 21 17:29:12.915332 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.915285    2563 scope.go:117] "RemoveContainer" containerID="f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"
Jul 21 17:29:12.915533 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.915516    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf"} err="failed to get container status \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": rpc error: code = NotFound desc = could not find container \"f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf\": container with ID starting with f1d5f0b28b16479087cafa1804cbcb7a3bc5c688e2dfec72705cb4881bf0c1cf not found: ID does not exist"
Jul 21 17:29:12.915587 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.915535    2563 scope.go:117] "RemoveContainer" containerID="090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"
Jul 21 17:29:12.915811 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.915772    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c"} err="failed to get container status \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": rpc error: code = NotFound desc = could not find container \"090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c\": container with ID starting with 090360cfb7f42d1316911fa7f7552a50930b561906accca52366a359da77dc6c not found: ID does not exist"
Jul 21 17:29:12.915891 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.915813    2563 scope.go:117] "RemoveContainer" containerID="8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"
Jul 21 17:29:12.916030 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.916014    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203"} err="failed to get container status \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": rpc error: code = NotFound desc = could not find container \"8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203\": container with ID starting with 8496ce941b71cf39dc963fec568905fd70687956b8ec05ab95f0088bead0c203 not found: ID does not exist"
Jul 21 17:29:12.916084 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.916029    2563 scope.go:117] "RemoveContainer" containerID="91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"
Jul 21 17:29:12.916250 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.916235    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f"} err="failed to get container status \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": rpc error: code = NotFound desc = could not find container \"91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f\": container with ID starting with 91932e2e08ebb3e47593e505cb50ef3a8a19b3b53f5b3338f73aec7fa8cc963f not found: ID does not exist"
Jul 21 17:29:12.916291 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.916250    2563 scope.go:117] "RemoveContainer" containerID="3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"
Jul 21 17:29:12.916512 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.916496    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c"} err="failed to get container status \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": rpc error: code = NotFound desc = could not find container \"3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c\": container with ID starting with 3c372e20f8ddd4cc537091b018fd0c0c010c4a2fed1ef71da8b1549e0e914c8c not found: ID does not exist"
Jul 21 17:29:12.916556 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.916501    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.919371 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.919351    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 21 17:29:12.919508 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.919492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 21 17:29:12.919581 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.919492    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-4fnb6prevqgk1\""
Jul 21 17:29:12.919581 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.919531    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 21 17:29:12.919660 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.919618    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 21 17:29:12.919885 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.919868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 21 17:29:12.919885 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.919877    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 21 17:29:12.919973 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.919937    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 21 17:29:12.920030 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.920003    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 21 17:29:12.920167 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.920155    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-ltcbj\""
Jul 21 17:29:12.920313 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.920207    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 21 17:29:12.920879 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.920860    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 21 17:29:12.920980 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.920868    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 21 17:29:12.921235 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.921219    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 21 17:29:12.923368 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.923349    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 21 17:29:12.926026 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.926010    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 21 17:29:12.930106 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.930088    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:29:12.998723 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.998691    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.998723 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.998721    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-config\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.998947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.998740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.998947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.998761    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.998947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.998776    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.998947 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.998850    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999164 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.998991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999164 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999164 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999164 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999164 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999160    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6545df95-7386-4091-b684-ac65fce2638e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999204    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999219    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999263    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhc8c\" (UniqueName: \"kubernetes.io/projected/6545df95-7386-4091-b684-ac65fce2638e-kube-api-access-lhc8c\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999315    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6545df95-7386-4091-b684-ac65fce2638e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999385 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999382    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:12.999703 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:12.999482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.099803 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.099803 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099805    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099831    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099846    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-config\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099925    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099950    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.099981    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100015 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100079    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6545df95-7386-4091-b684-ac65fce2638e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100111    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100162    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100187    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100214    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhc8c\" (UniqueName: \"kubernetes.io/projected/6545df95-7386-4091-b684-ac65fce2638e-kube-api-access-lhc8c\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100258    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.100379 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100287    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6545df95-7386-4091-b684-ac65fce2638e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.101036 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.100969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.101573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.101546    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.102863 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.102713    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.104108 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.103086    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/6545df95-7386-4091-b684-ac65fce2638e-config-out\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.104108 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.103089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.104108 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.103309    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-web-config\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.104108 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.103452    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.104108 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.103860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.104108 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.103969    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.104434 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.104365    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.104434 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.104395    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.105053 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.104588    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.105053 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.104613    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-config\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.105053 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.105016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.105053 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.104974    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/6545df95-7386-4091-b684-ac65fce2638e-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.105486 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.105466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.106354 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.106334    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.106632 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.106615    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/6545df95-7386-4091-b684-ac65fce2638e-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.106853 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.106837    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/6545df95-7386-4091-b684-ac65fce2638e-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.114977 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.114958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhc8c\" (UniqueName: \"kubernetes.io/projected/6545df95-7386-4091-b684-ac65fce2638e-kube-api-access-lhc8c\") pod \"prometheus-k8s-0\" (UID: \"6545df95-7386-4091-b684-ac65fce2638e\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.227047 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.226978    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:29:13.366939 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.366915    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 21 17:29:13.369493 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:29:13.369461    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6545df95_7386_4091_b684_ac65fce2638e.slice/crio-016c7183998bb2fb51a17bbeb1f22a0ae98269381fe19f99b5d5709918a129ce WatchSource:0}: Error finding container 016c7183998bb2fb51a17bbeb1f22a0ae98269381fe19f99b5d5709918a129ce: Status 404 returned error can't find the container with id 016c7183998bb2fb51a17bbeb1f22a0ae98269381fe19f99b5d5709918a129ce
Jul 21 17:29:13.858685 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.858645    2563 generic.go:356] "Generic (PLEG): container finished" podID="6545df95-7386-4091-b684-ac65fce2638e" containerID="e8c553b657d1e4e836e46944b0d6e3dad5877a8e17f747dbfa3375b1bb156b47" exitCode=0
Jul 21 17:29:13.859148 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.858744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545df95-7386-4091-b684-ac65fce2638e","Type":"ContainerDied","Data":"e8c553b657d1e4e836e46944b0d6e3dad5877a8e17f747dbfa3375b1bb156b47"}
Jul 21 17:29:13.859148 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:13.858847    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545df95-7386-4091-b684-ac65fce2638e","Type":"ContainerStarted","Data":"016c7183998bb2fb51a17bbeb1f22a0ae98269381fe19f99b5d5709918a129ce"}
Jul 21 17:29:14.867275 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:14.867237    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545df95-7386-4091-b684-ac65fce2638e","Type":"ContainerStarted","Data":"24b19230fb566d413741b26b13dc4cf7266e930d2abd5c01f2853c37a36071fb"}
Jul 21 17:29:14.867638 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:14.867301    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545df95-7386-4091-b684-ac65fce2638e","Type":"ContainerStarted","Data":"69a4ae4664a098a27ce0e27f631677fd9564e757c260151380c277d45eb30d78"}
Jul 21 17:29:14.867638 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:14.867327    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545df95-7386-4091-b684-ac65fce2638e","Type":"ContainerStarted","Data":"f1ba9c35cfba4a21138030dca6b4471997191bea3ac392de78e6e4091749508c"}
Jul 21 17:29:14.867638 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:14.867351    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545df95-7386-4091-b684-ac65fce2638e","Type":"ContainerStarted","Data":"b48ac7b44809528671460ce855ca6489f4e2b08e9656bbde292832d293515f99"}
Jul 21 17:29:14.867638 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:14.867372    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545df95-7386-4091-b684-ac65fce2638e","Type":"ContainerStarted","Data":"65ebe034d62c57e90e4a6335caa22ab5ee1991cdbb12bea9b78e9e89dd1dc4ac"}
Jul 21 17:29:14.867638 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:14.867394    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"6545df95-7386-4091-b684-ac65fce2638e","Type":"ContainerStarted","Data":"2de889854e762f580edcf37a729d11d85f27e585b505f221779f892491905704"}
Jul 21 17:29:14.907658 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:14.907525    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.888531425 podStartE2EDuration="2.907505781s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.974356ms" isStatefulPod=false podCreationTimestamp="2026-07-21 17:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:29:14.905350556 +0000 UTC m=+214.463357664" watchObservedRunningTime="2026-07-21 17:29:14.907505781 +0000 UTC m=+214.465512889"
Jul 21 17:29:18.227737 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:29:18.227703    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:30:13.227570 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:30:13.227487    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:30:13.243747 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:30:13.243723    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:30:14.124352 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:30:14.124318    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 21 17:30:40.778748 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:30:40.778716    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:30:40.786094 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:30:40.786070    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 21 17:30:40.786369 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:30:40.786355    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:33:50.628486 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.628452    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-rrnnb"]
Jul 21 17:33:50.631827 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.631810    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rrnnb"
Jul 21 17:33:50.635097 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.635079    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 21 17:33:50.635201 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.635170    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kwgsb\""
Jul 21 17:33:50.636433 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.636416    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 21 17:33:50.636433 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.636424    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 21 17:33:50.643925 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.643904    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rrnnb"]
Jul 21 17:33:50.710408 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.710377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-85942\" (UniqueName: \"kubernetes.io/projected/ee5c1828-9768-4bd9-833f-548609d7b59e-kube-api-access-85942\") pod \"s3-init-rrnnb\" (UID: \"ee5c1828-9768-4bd9-833f-548609d7b59e\") " pod="kserve/s3-init-rrnnb"
Jul 21 17:33:50.710408 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.710411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5c1828-9768-4bd9-833f-548609d7b59e-model-cache\") pod \"s3-init-rrnnb\" (UID: \"ee5c1828-9768-4bd9-833f-548609d7b59e\") " pod="kserve/s3-init-rrnnb"
Jul 21 17:33:50.811055 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.811008    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-85942\" (UniqueName: \"kubernetes.io/projected/ee5c1828-9768-4bd9-833f-548609d7b59e-kube-api-access-85942\") pod \"s3-init-rrnnb\" (UID: \"ee5c1828-9768-4bd9-833f-548609d7b59e\") " pod="kserve/s3-init-rrnnb"
Jul 21 17:33:50.811055 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.811056    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5c1828-9768-4bd9-833f-548609d7b59e-model-cache\") pod \"s3-init-rrnnb\" (UID: \"ee5c1828-9768-4bd9-833f-548609d7b59e\") " pod="kserve/s3-init-rrnnb"
Jul 21 17:33:50.811401 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.811386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5c1828-9768-4bd9-833f-548609d7b59e-model-cache\") pod \"s3-init-rrnnb\" (UID: \"ee5c1828-9768-4bd9-833f-548609d7b59e\") " pod="kserve/s3-init-rrnnb"
Jul 21 17:33:50.819696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.819672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-85942\" (UniqueName: \"kubernetes.io/projected/ee5c1828-9768-4bd9-833f-548609d7b59e-kube-api-access-85942\") pod \"s3-init-rrnnb\" (UID: \"ee5c1828-9768-4bd9-833f-548609d7b59e\") " pod="kserve/s3-init-rrnnb"
Jul 21 17:33:50.954100 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:50.954026    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rrnnb"
Jul 21 17:33:51.106371 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:51.106275    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-rrnnb"]
Jul 21 17:33:51.109062 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:33:51.109030    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee5c1828_9768_4bd9_833f_548609d7b59e.slice/crio-006beed333e50a146f4bb35396368fc1167433ac11f9c15913c7f9731289a314 WatchSource:0}: Error finding container 006beed333e50a146f4bb35396368fc1167433ac11f9c15913c7f9731289a314: Status 404 returned error can't find the container with id 006beed333e50a146f4bb35396368fc1167433ac11f9c15913c7f9731289a314
Jul 21 17:33:51.111072 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:51.111052    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:33:51.987984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:51.987945    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rrnnb" event={"ID":"ee5c1828-9768-4bd9-833f-548609d7b59e","Type":"ContainerStarted","Data":"006beed333e50a146f4bb35396368fc1167433ac11f9c15913c7f9731289a314"}
Jul 21 17:33:56.012071 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:56.012030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rrnnb" event={"ID":"ee5c1828-9768-4bd9-833f-548609d7b59e","Type":"ContainerStarted","Data":"11661e1a2f44d583dea80fb83781023a7d26260b6de0c0aa42692ef58e639381"}
Jul 21 17:33:58.021763 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:58.021733    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee5c1828-9768-4bd9-833f-548609d7b59e" containerID="11661e1a2f44d583dea80fb83781023a7d26260b6de0c0aa42692ef58e639381" exitCode=0
Jul 21 17:33:58.022152 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:33:58.021776    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rrnnb" event={"ID":"ee5c1828-9768-4bd9-833f-548609d7b59e","Type":"ContainerDied","Data":"11661e1a2f44d583dea80fb83781023a7d26260b6de0c0aa42692ef58e639381"}
Jul 21 17:34:03.049168 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:34:03.049135    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rrnnb" event={"ID":"ee5c1828-9768-4bd9-833f-548609d7b59e","Type":"ContainerStarted","Data":"cc0abbc11be38a2d73628126fb38dc898d02ce97afa758c657dd0b9a0ad78980"}
Jul 21 17:34:03.087297 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:34:03.087247    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-rrnnb" podStartSLOduration=2.42243807 podStartE2EDuration="13.087231978s" totalImagesPullingTime="8.887639834s" totalInitContainerRuntime="1.777154074s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:33:50 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:34:03.086076042 +0000 UTC m=+502.644083151" watchObservedRunningTime="2026-07-21 17:34:03.087231978 +0000 UTC m=+502.645239085"
Jul 21 17:35:07.309396 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:07.309360    2563 generic.go:356] "Generic (PLEG): container finished" podID="ee5c1828-9768-4bd9-833f-548609d7b59e" containerID="cc0abbc11be38a2d73628126fb38dc898d02ce97afa758c657dd0b9a0ad78980" exitCode=0
Jul 21 17:35:07.309766 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:07.309410    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rrnnb" event={"ID":"ee5c1828-9768-4bd9-833f-548609d7b59e","Type":"ContainerDied","Data":"cc0abbc11be38a2d73628126fb38dc898d02ce97afa758c657dd0b9a0ad78980"}
Jul 21 17:35:08.438325 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:08.438303    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rrnnb"
Jul 21 17:35:08.582004 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:08.581918    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-85942\" (UniqueName: \"kubernetes.io/projected/ee5c1828-9768-4bd9-833f-548609d7b59e-kube-api-access-85942\") pod \"ee5c1828-9768-4bd9-833f-548609d7b59e\" (UID: \"ee5c1828-9768-4bd9-833f-548609d7b59e\") "
Jul 21 17:35:08.582173 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:08.582028    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5c1828-9768-4bd9-833f-548609d7b59e-model-cache\") pod \"ee5c1828-9768-4bd9-833f-548609d7b59e\" (UID: \"ee5c1828-9768-4bd9-833f-548609d7b59e\") "
Jul 21 17:35:08.584378 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:08.584349    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5c1828-9768-4bd9-833f-548609d7b59e-kube-api-access-85942" (OuterVolumeSpecName: "kube-api-access-85942") pod "ee5c1828-9768-4bd9-833f-548609d7b59e" (UID: "ee5c1828-9768-4bd9-833f-548609d7b59e"). InnerVolumeSpecName "kube-api-access-85942". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:35:08.600910 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:08.600885    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5c1828-9768-4bd9-833f-548609d7b59e-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee5c1828-9768-4bd9-833f-548609d7b59e" (UID: "ee5c1828-9768-4bd9-833f-548609d7b59e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:35:08.683245 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:08.683218    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5c1828-9768-4bd9-833f-548609d7b59e-model-cache\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:35:08.683245 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:08.683242    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-85942\" (UniqueName: \"kubernetes.io/projected/ee5c1828-9768-4bd9-833f-548609d7b59e-kube-api-access-85942\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:35:09.318613 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:09.318579    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-rrnnb" event={"ID":"ee5c1828-9768-4bd9-833f-548609d7b59e","Type":"ContainerDied","Data":"006beed333e50a146f4bb35396368fc1167433ac11f9c15913c7f9731289a314"}
Jul 21 17:35:09.318613 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:09.318598    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-rrnnb"
Jul 21 17:35:09.318613 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:09.318619    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="006beed333e50a146f4bb35396368fc1167433ac11f9c15913c7f9731289a314"
Jul 21 17:35:16.599524 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.599493    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-s7bk8"]
Jul 21 17:35:16.599984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.599868    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5c1828-9768-4bd9-833f-548609d7b59e" containerName="download-hf-model"
Jul 21 17:35:16.599984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.599879    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5c1828-9768-4bd9-833f-548609d7b59e" containerName="download-hf-model"
Jul 21 17:35:16.599984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.599888    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5c1828-9768-4bd9-833f-548609d7b59e" containerName="s3-init"
Jul 21 17:35:16.599984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.599893    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5c1828-9768-4bd9-833f-548609d7b59e" containerName="s3-init"
Jul 21 17:35:16.599984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.599944    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5c1828-9768-4bd9-833f-548609d7b59e" containerName="s3-init"
Jul 21 17:35:16.603055 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.603037    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:16.605534 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.605513    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 21 17:35:16.605673 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.605563    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 21 17:35:16.605673 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.605608    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kwgsb\""
Jul 21 17:35:16.606957 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.606940    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 21 17:35:16.611756 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.611735    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-s7bk8"]
Jul 21 17:35:16.646842 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.646823    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-model-cache\") pod \"s3-tls-init-custom-s7bk8\" (UID: \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\") " pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:16.646974 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.646876    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wphnb\" (UniqueName: \"kubernetes.io/projected/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-kube-api-access-wphnb\") pod \"s3-tls-init-custom-s7bk8\" (UID: \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\") " pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:16.747530 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.747506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wphnb\" (UniqueName: \"kubernetes.io/projected/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-kube-api-access-wphnb\") pod \"s3-tls-init-custom-s7bk8\" (UID: \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\") " pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:16.747631 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.747560    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-model-cache\") pod \"s3-tls-init-custom-s7bk8\" (UID: \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\") " pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:16.747884 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.747869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-model-cache\") pod \"s3-tls-init-custom-s7bk8\" (UID: \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\") " pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:16.756696 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.756674    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wphnb\" (UniqueName: \"kubernetes.io/projected/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-kube-api-access-wphnb\") pod \"s3-tls-init-custom-s7bk8\" (UID: \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\") " pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:16.922917 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:16.922849    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:17.056996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:17.056972    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-s7bk8"]
Jul 21 17:35:17.059119 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:35:17.059085    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6cbaa73b_d112_4a01_8ef7_4217d26cebc1.slice/crio-5942707134e58abe8a83b333ff1092bb37a4f24d043edec282d2b25f1f1be75a WatchSource:0}: Error finding container 5942707134e58abe8a83b333ff1092bb37a4f24d043edec282d2b25f1f1be75a: Status 404 returned error can't find the container with id 5942707134e58abe8a83b333ff1092bb37a4f24d043edec282d2b25f1f1be75a
Jul 21 17:35:17.354909 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:17.354855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s7bk8" event={"ID":"6cbaa73b-d112-4a01-8ef7-4217d26cebc1","Type":"ContainerStarted","Data":"a2f259983f463e0041f68a2a47362202a5adb202707e8b45f0db8e04d0b23a67"}
Jul 21 17:35:17.355091 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:17.354920    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s7bk8" event={"ID":"6cbaa73b-d112-4a01-8ef7-4217d26cebc1","Type":"ContainerStarted","Data":"5942707134e58abe8a83b333ff1092bb37a4f24d043edec282d2b25f1f1be75a"}
Jul 21 17:35:19.365051 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:19.364963    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cbaa73b-d112-4a01-8ef7-4217d26cebc1" containerID="a2f259983f463e0041f68a2a47362202a5adb202707e8b45f0db8e04d0b23a67" exitCode=0
Jul 21 17:35:19.365392 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:19.365043    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s7bk8" event={"ID":"6cbaa73b-d112-4a01-8ef7-4217d26cebc1","Type":"ContainerDied","Data":"a2f259983f463e0041f68a2a47362202a5adb202707e8b45f0db8e04d0b23a67"}
Jul 21 17:35:20.371292 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:20.371257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s7bk8" event={"ID":"6cbaa73b-d112-4a01-8ef7-4217d26cebc1","Type":"ContainerStarted","Data":"4331621ffbfc315bd2690e6be7c04394c543d11a0390006cf8f52135452c0eb0"}
Jul 21 17:35:20.392509 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:20.392461    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-s7bk8" podStartSLOduration=2.48196008 podStartE2EDuration="4.392445836s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.910485756s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:35:20.391652987 +0000 UTC m=+579.949660094" watchObservedRunningTime="2026-07-21 17:35:20.392445836 +0000 UTC m=+579.950452943"
Jul 21 17:35:26.398027 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:26.397992    2563 generic.go:356] "Generic (PLEG): container finished" podID="6cbaa73b-d112-4a01-8ef7-4217d26cebc1" containerID="4331621ffbfc315bd2690e6be7c04394c543d11a0390006cf8f52135452c0eb0" exitCode=0
Jul 21 17:35:26.398371 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:26.398039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s7bk8" event={"ID":"6cbaa73b-d112-4a01-8ef7-4217d26cebc1","Type":"ContainerDied","Data":"4331621ffbfc315bd2690e6be7c04394c543d11a0390006cf8f52135452c0eb0"}
Jul 21 17:35:27.544770 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:27.544745    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:27.644287 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:27.644246    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-model-cache\") pod \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\" (UID: \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\") "
Jul 21 17:35:27.644443 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:27.644322    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wphnb\" (UniqueName: \"kubernetes.io/projected/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-kube-api-access-wphnb\") pod \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\" (UID: \"6cbaa73b-d112-4a01-8ef7-4217d26cebc1\") "
Jul 21 17:35:27.646392 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:27.646370    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-kube-api-access-wphnb" (OuterVolumeSpecName: "kube-api-access-wphnb") pod "6cbaa73b-d112-4a01-8ef7-4217d26cebc1" (UID: "6cbaa73b-d112-4a01-8ef7-4217d26cebc1"). InnerVolumeSpecName "kube-api-access-wphnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:35:27.667562 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:27.667506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-model-cache" (OuterVolumeSpecName: "model-cache") pod "6cbaa73b-d112-4a01-8ef7-4217d26cebc1" (UID: "6cbaa73b-d112-4a01-8ef7-4217d26cebc1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:35:27.745776 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:27.745751    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-model-cache\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:35:27.745776 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:27.745773    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wphnb\" (UniqueName: \"kubernetes.io/projected/6cbaa73b-d112-4a01-8ef7-4217d26cebc1-kube-api-access-wphnb\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:35:28.407610 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:28.407584    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-s7bk8"
Jul 21 17:35:28.407610 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:28.407598    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-s7bk8" event={"ID":"6cbaa73b-d112-4a01-8ef7-4217d26cebc1","Type":"ContainerDied","Data":"5942707134e58abe8a83b333ff1092bb37a4f24d043edec282d2b25f1f1be75a"}
Jul 21 17:35:28.407830 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:28.407629    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5942707134e58abe8a83b333ff1092bb37a4f24d043edec282d2b25f1f1be75a"
Jul 21 17:35:33.811176 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.811139    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-92qlj"]
Jul 21 17:35:33.811560 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.811466    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cbaa73b-d112-4a01-8ef7-4217d26cebc1" containerName="download-hf-model"
Jul 21 17:35:33.811560 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.811477    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cbaa73b-d112-4a01-8ef7-4217d26cebc1" containerName="download-hf-model"
Jul 21 17:35:33.811560 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.811493    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6cbaa73b-d112-4a01-8ef7-4217d26cebc1" containerName="s3-tls-init-custom"
Jul 21 17:35:33.811560 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.811498    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6cbaa73b-d112-4a01-8ef7-4217d26cebc1" containerName="s3-tls-init-custom"
Jul 21 17:35:33.811560 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.811544    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6cbaa73b-d112-4a01-8ef7-4217d26cebc1" containerName="s3-tls-init-custom"
Jul 21 17:35:33.814491 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.814473    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:33.817215 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.817152    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 21 17:35:33.817215 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.817194    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 21 17:35:33.818461 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.818444    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 21 17:35:33.818573 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.818501    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-kwgsb\""
Jul 21 17:35:33.822987 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.822966    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-92qlj"]
Jul 21 17:35:33.899232 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.899190    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-842fr\" (UniqueName: \"kubernetes.io/projected/4d270610-b9ca-435c-80c6-a8e3456c0886-kube-api-access-842fr\") pod \"s3-tls-init-serving-92qlj\" (UID: \"4d270610-b9ca-435c-80c6-a8e3456c0886\") " pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:33.899232 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:33.899242    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d270610-b9ca-435c-80c6-a8e3456c0886-model-cache\") pod \"s3-tls-init-serving-92qlj\" (UID: \"4d270610-b9ca-435c-80c6-a8e3456c0886\") " pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:34.000397 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:34.000341    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d270610-b9ca-435c-80c6-a8e3456c0886-model-cache\") pod \"s3-tls-init-serving-92qlj\" (UID: \"4d270610-b9ca-435c-80c6-a8e3456c0886\") " pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:34.000597 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:34.000465    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-842fr\" (UniqueName: \"kubernetes.io/projected/4d270610-b9ca-435c-80c6-a8e3456c0886-kube-api-access-842fr\") pod \"s3-tls-init-serving-92qlj\" (UID: \"4d270610-b9ca-435c-80c6-a8e3456c0886\") " pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:34.000743 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:34.000719    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d270610-b9ca-435c-80c6-a8e3456c0886-model-cache\") pod \"s3-tls-init-serving-92qlj\" (UID: \"4d270610-b9ca-435c-80c6-a8e3456c0886\") " pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:34.009324 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:34.009293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-842fr\" (UniqueName: \"kubernetes.io/projected/4d270610-b9ca-435c-80c6-a8e3456c0886-kube-api-access-842fr\") pod \"s3-tls-init-serving-92qlj\" (UID: \"4d270610-b9ca-435c-80c6-a8e3456c0886\") " pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:34.131550 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:34.131472    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:34.263826 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:34.263802    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-92qlj"]
Jul 21 17:35:34.266274 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:35:34.266246    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4d270610_b9ca_435c_80c6_a8e3456c0886.slice/crio-106f8e2be1f47a4d2e8f52800b278d874b345ad65e299205e1fb585da01d6ed7 WatchSource:0}: Error finding container 106f8e2be1f47a4d2e8f52800b278d874b345ad65e299205e1fb585da01d6ed7: Status 404 returned error can't find the container with id 106f8e2be1f47a4d2e8f52800b278d874b345ad65e299205e1fb585da01d6ed7
Jul 21 17:35:34.433486 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:34.433396    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-92qlj" event={"ID":"4d270610-b9ca-435c-80c6-a8e3456c0886","Type":"ContainerStarted","Data":"181f1b62dce1faa41d244d9cf1f1d805e4821b4c28335e9facfc0c35a6aab7aa"}
Jul 21 17:35:34.433486 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:34.433442    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-92qlj" event={"ID":"4d270610-b9ca-435c-80c6-a8e3456c0886","Type":"ContainerStarted","Data":"106f8e2be1f47a4d2e8f52800b278d874b345ad65e299205e1fb585da01d6ed7"}
Jul 21 17:35:36.444021 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:36.443995    2563 generic.go:356] "Generic (PLEG): container finished" podID="4d270610-b9ca-435c-80c6-a8e3456c0886" containerID="181f1b62dce1faa41d244d9cf1f1d805e4821b4c28335e9facfc0c35a6aab7aa" exitCode=0
Jul 21 17:35:36.444301 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:36.444064    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-92qlj" event={"ID":"4d270610-b9ca-435c-80c6-a8e3456c0886","Type":"ContainerDied","Data":"181f1b62dce1faa41d244d9cf1f1d805e4821b4c28335e9facfc0c35a6aab7aa"}
Jul 21 17:35:37.450643 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:37.450610    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-92qlj" event={"ID":"4d270610-b9ca-435c-80c6-a8e3456c0886","Type":"ContainerStarted","Data":"198e2176dae26fd8285c49d6225e90bb1f3215a0142a8e03cc1eab30781ceab0"}
Jul 21 17:35:37.473126 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:37.473081    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-92qlj" podStartSLOduration=2.374110631 podStartE2EDuration="4.473066429s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.098955798s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:35:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:35:37.47076611 +0000 UTC m=+597.028773216" watchObservedRunningTime="2026-07-21 17:35:37.473066429 +0000 UTC m=+597.031073535"
Jul 21 17:35:40.810912 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:40.810875    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:35:40.816171 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:40.816145    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:35:43.482498 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:43.482410    2563 generic.go:356] "Generic (PLEG): container finished" podID="4d270610-b9ca-435c-80c6-a8e3456c0886" containerID="198e2176dae26fd8285c49d6225e90bb1f3215a0142a8e03cc1eab30781ceab0" exitCode=0
Jul 21 17:35:43.482856 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:43.482489    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-92qlj" event={"ID":"4d270610-b9ca-435c-80c6-a8e3456c0886","Type":"ContainerDied","Data":"198e2176dae26fd8285c49d6225e90bb1f3215a0142a8e03cc1eab30781ceab0"}
Jul 21 17:35:44.632236 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:44.632212    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:44.791640 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:44.791611    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d270610-b9ca-435c-80c6-a8e3456c0886-model-cache\") pod \"4d270610-b9ca-435c-80c6-a8e3456c0886\" (UID: \"4d270610-b9ca-435c-80c6-a8e3456c0886\") "
Jul 21 17:35:44.791830 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:44.791704    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-842fr\" (UniqueName: \"kubernetes.io/projected/4d270610-b9ca-435c-80c6-a8e3456c0886-kube-api-access-842fr\") pod \"4d270610-b9ca-435c-80c6-a8e3456c0886\" (UID: \"4d270610-b9ca-435c-80c6-a8e3456c0886\") "
Jul 21 17:35:44.794001 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:44.793978    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d270610-b9ca-435c-80c6-a8e3456c0886-kube-api-access-842fr" (OuterVolumeSpecName: "kube-api-access-842fr") pod "4d270610-b9ca-435c-80c6-a8e3456c0886" (UID: "4d270610-b9ca-435c-80c6-a8e3456c0886"). InnerVolumeSpecName "kube-api-access-842fr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:35:44.814632 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:44.814602    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4d270610-b9ca-435c-80c6-a8e3456c0886-model-cache" (OuterVolumeSpecName: "model-cache") pod "4d270610-b9ca-435c-80c6-a8e3456c0886" (UID: "4d270610-b9ca-435c-80c6-a8e3456c0886"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:35:44.893252 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:44.893218    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4d270610-b9ca-435c-80c6-a8e3456c0886-model-cache\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:35:44.893252 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:44.893245    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-842fr\" (UniqueName: \"kubernetes.io/projected/4d270610-b9ca-435c-80c6-a8e3456c0886-kube-api-access-842fr\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:35:45.495048 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:45.495014    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-92qlj"
Jul 21 17:35:45.495195 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:45.495048    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-92qlj" event={"ID":"4d270610-b9ca-435c-80c6-a8e3456c0886","Type":"ContainerDied","Data":"106f8e2be1f47a4d2e8f52800b278d874b345ad65e299205e1fb585da01d6ed7"}
Jul 21 17:35:45.495195 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:35:45.495083    2563 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="106f8e2be1f47a4d2e8f52800b278d874b345ad65e299205e1fb585da01d6ed7"
Jul 21 17:39:17.728105 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.728026    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"]
Jul 21 17:39:17.728550 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.728367    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d270610-b9ca-435c-80c6-a8e3456c0886" containerName="s3-tls-init-serving"
Jul 21 17:39:17.728550 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.728379    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d270610-b9ca-435c-80c6-a8e3456c0886" containerName="s3-tls-init-serving"
Jul 21 17:39:17.728550 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.728399    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d270610-b9ca-435c-80c6-a8e3456c0886" containerName="download-hf-model"
Jul 21 17:39:17.728550 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.728409    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d270610-b9ca-435c-80c6-a8e3456c0886" containerName="download-hf-model"
Jul 21 17:39:17.728550 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.728479    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d270610-b9ca-435c-80c6-a8e3456c0886" containerName="s3-tls-init-serving"
Jul 21 17:39:17.731594 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.731578    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:17.734323 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.734302    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-kube-rbac-proxy-sar-config\""
Jul 21 17:39:17.734429 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.734394    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:39:17.734429 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.734415    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:39:17.734804 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.734779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"message-dumper-predictor-serving-cert\""
Jul 21 17:39:17.735845 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.735832    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:39:17.740784 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.740762    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"]
Jul 21 17:39:17.903291 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.903256    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:17.903474 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.903318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:17.903474 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:17.903343    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98x8n\" (UniqueName: \"kubernetes.io/projected/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-api-access-98x8n\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:18.003830 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.003768    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:18.004013 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.003864    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:18.004013 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.003888    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98x8n\" (UniqueName: \"kubernetes.io/projected/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-api-access-98x8n\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:18.004559 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.004538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-rbac-proxy-sar-config\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:18.006310 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.006292    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-proxy-tls\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:18.013512 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.013491    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-98x8n\" (UniqueName: \"kubernetes.io/projected/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-api-access-98x8n\") pod \"message-dumper-predictor-6575bcfdfd-9kbll\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") " pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:18.042449 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.042416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:18.175998 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.175975    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"]
Jul 21 17:39:18.178216 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:39:18.178193    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e95155a_51d9_4d12_8ac3_7c1fbc50a9cb.slice/crio-617f54ecd8e7ce89ae40966161e80a6651c51213b7d59354a83e1a6ec8076b39 WatchSource:0}: Error finding container 617f54ecd8e7ce89ae40966161e80a6651c51213b7d59354a83e1a6ec8076b39: Status 404 returned error can't find the container with id 617f54ecd8e7ce89ae40966161e80a6651c51213b7d59354a83e1a6ec8076b39
Jul 21 17:39:18.180179 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.180159    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:39:18.347598 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:18.347515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" event={"ID":"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb","Type":"ContainerStarted","Data":"617f54ecd8e7ce89ae40966161e80a6651c51213b7d59354a83e1a6ec8076b39"}
Jul 21 17:39:20.359579 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:20.359541    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" event={"ID":"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb","Type":"ContainerStarted","Data":"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3"}
Jul 21 17:39:22.371241 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:22.371150    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" event={"ID":"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb","Type":"ContainerStarted","Data":"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340"}
Jul 21 17:39:22.371653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:22.371253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:22.371653 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:22.371287    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:22.372962 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:22.372940    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:39:22.391423 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:22.391378    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" podStartSLOduration=1.5198391230000001 podStartE2EDuration="5.391365042s" totalImagesPullingTime="3.871525919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:39:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:39:22.389186738 +0000 UTC m=+821.947193845" watchObservedRunningTime="2026-07-21 17:39:22.391365042 +0000 UTC m=+821.949372149"
Jul 21 17:39:29.387386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:39:29.387354    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:40:40.844120 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:40:40.844088    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:40:40.845353 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:40:40.845322    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:41:12.813401 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:12.813375    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_message-dumper-predictor-6575bcfdfd-9kbll_6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb/kserve-container/0.log"
Jul 21 17:41:13.173324 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.173251    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"]
Jul 21 17:41:13.173511 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.173489    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerName="kserve-container" containerID="cri-o://d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3" gracePeriod=30
Jul 21 17:41:13.173554 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.173501    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerName="kube-rbac-proxy" containerID="cri-o://8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340" gracePeriod=30
Jul 21 17:41:13.418134 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.418111    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:41:13.484710 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.484638    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-proxy-tls\") pod \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") "
Jul 21 17:41:13.484710 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.484687    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-rbac-proxy-sar-config\") pod \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") "
Jul 21 17:41:13.484965 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.484816    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-98x8n\" (UniqueName: \"kubernetes.io/projected/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-api-access-98x8n\") pod \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\" (UID: \"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb\") "
Jul 21 17:41:13.485049 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.485028    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" (UID: "6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:41:13.485124 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.485111    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:41:13.486939 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.486907    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" (UID: "6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:41:13.487013 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.486987    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-api-access-98x8n" (OuterVolumeSpecName: "kube-api-access-98x8n") pod "6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" (UID: "6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb"). InnerVolumeSpecName "kube-api-access-98x8n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:41:13.586322 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.586294    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:41:13.586322 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.586319    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98x8n\" (UniqueName: \"kubernetes.io/projected/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb-kube-api-access-98x8n\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:41:13.835808 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.835755    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerID="8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340" exitCode=2
Jul 21 17:41:13.835808 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.835778    2563 generic.go:356] "Generic (PLEG): container finished" podID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerID="d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3" exitCode=2
Jul 21 17:41:13.836329 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.835839    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"
Jul 21 17:41:13.836329 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.835838    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" event={"ID":"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb","Type":"ContainerDied","Data":"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340"}
Jul 21 17:41:13.836329 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.835903    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" event={"ID":"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb","Type":"ContainerDied","Data":"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3"}
Jul 21 17:41:13.836329 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.835924    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll" event={"ID":"6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb","Type":"ContainerDied","Data":"617f54ecd8e7ce89ae40966161e80a6651c51213b7d59354a83e1a6ec8076b39"}
Jul 21 17:41:13.836329 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.835949    2563 scope.go:117] "RemoveContainer" containerID="8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340"
Jul 21 17:41:13.845066 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.845049    2563 scope.go:117] "RemoveContainer" containerID="d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3"
Jul 21 17:41:13.853175 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.853158    2563 scope.go:117] "RemoveContainer" containerID="8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340"
Jul 21 17:41:13.853425 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:41:13.853403    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340\": container with ID starting with 8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340 not found: ID does not exist" containerID="8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340"
Jul 21 17:41:13.853559 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.853434    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340"} err="failed to get container status \"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340\": rpc error: code = NotFound desc = could not find container \"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340\": container with ID starting with 8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340 not found: ID does not exist"
Jul 21 17:41:13.853559 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.853459    2563 scope.go:117] "RemoveContainer" containerID="d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3"
Jul 21 17:41:13.853714 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:41:13.853697    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3\": container with ID starting with d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3 not found: ID does not exist" containerID="d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3"
Jul 21 17:41:13.853752 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.853721    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3"} err="failed to get container status \"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3\": rpc error: code = NotFound desc = could not find container \"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3\": container with ID starting with d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3 not found: ID does not exist"
Jul 21 17:41:13.853752 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.853748    2563 scope.go:117] "RemoveContainer" containerID="8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340"
Jul 21 17:41:13.854007 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.853988    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340"} err="failed to get container status \"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340\": rpc error: code = NotFound desc = could not find container \"8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340\": container with ID starting with 8f3729ebbc646a5c4f16003489d23fef4e186f4ee9363133a984720092b37340 not found: ID does not exist"
Jul 21 17:41:13.854054 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.854008    2563 scope.go:117] "RemoveContainer" containerID="d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3"
Jul 21 17:41:13.854224 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.854206    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3"} err="failed to get container status \"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3\": rpc error: code = NotFound desc = could not find container \"d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3\": container with ID starting with d004ad2a65b3704298d11cc8279b72ff2a4c6d7613b8a5069a35eed85ff092e3 not found: ID does not exist"
Jul 21 17:41:13.857830 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.857782    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"]
Jul 21 17:41:13.861703 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:13.861682    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/message-dumper-predictor-6575bcfdfd-9kbll"]
Jul 21 17:41:14.908684 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:41:14.908643    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" path="/var/lib/kubelet/pods/6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb/volumes"
Jul 21 17:45:40.878365 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:45:40.878335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:45:40.878865 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:45:40.878335    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:46:46.151083 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.151051    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"]
Jul 21 17:46:46.153386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.151400    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerName="kserve-container"
Jul 21 17:46:46.153386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.151412    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerName="kserve-container"
Jul 21 17:46:46.153386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.151432    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerName="kube-rbac-proxy"
Jul 21 17:46:46.153386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.151440    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerName="kube-rbac-proxy"
Jul 21 17:46:46.153386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.151494    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerName="kserve-container"
Jul 21 17:46:46.153386 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.151504    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e95155a-51d9-4d12-8ac3-7c1fbc50a9cb" containerName="kube-rbac-proxy"
Jul 21 17:46:46.154431 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.154416    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.157120 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.157098    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-predictor-serving-cert\""
Jul 21 17:46:46.157120 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.157114    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:46:46.157297 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.157107    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:46:46.157297 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.157157    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:46:46.157297 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.157100    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-v2-variants-kube-rbac-proxy-sar-config\""
Jul 21 17:46:46.158198 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.158184    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:46:46.158254 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.158191    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:46:46.169485 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.169465    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"]
Jul 21 17:46:46.225411 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.225372    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.225586 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.225441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.225586 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.225500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.225586 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.225521    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.225586 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.225547    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts24s\" (UniqueName: \"kubernetes.io/projected/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-api-access-ts24s\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.326733 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.326697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.326905 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.326744    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.326905 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.326775    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.326905 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.326814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.326905 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.326847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts24s\" (UniqueName: \"kubernetes.io/projected/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-api-access-ts24s\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.327167 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.327137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kserve-provision-location\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.327546 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.327525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-cabundle-cert\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.327589 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.327525    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.329538 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.329516    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-proxy-tls\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.334984 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.334952    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts24s\" (UniqueName: \"kubernetes.io/projected/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-api-access-ts24s\") pod \"isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") " pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.464580 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.464490    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:46:46.602617 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:46.602537    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"]
Jul 21 17:46:46.604902 ip-10-0-129-22 kubenswrapper[2563]: W0721 17:46:46.604873    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65d9cf7c_7229_49ef_8ca9_15e83f98e6ce.slice/crio-6cd43b1f5efd82d88d010576d2d781e47d2bf9c54d67507862fd68d56697c20f WatchSource:0}: Error finding container 6cd43b1f5efd82d88d010576d2d781e47d2bf9c54d67507862fd68d56697c20f: Status 404 returned error can't find the container with id 6cd43b1f5efd82d88d010576d2d781e47d2bf9c54d67507862fd68d56697c20f
Jul 21 17:46:47.222423 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:47.222387    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" event={"ID":"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce","Type":"ContainerStarted","Data":"67c23b4556a95baf66e5b72fc06da20bd865d95d2bd8cc61a4d1e50b0ba6de4f"}
Jul 21 17:46:47.222848 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:46:47.222458    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" event={"ID":"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce","Type":"ContainerStarted","Data":"6cd43b1f5efd82d88d010576d2d781e47d2bf9c54d67507862fd68d56697c20f"}
Jul 21 17:47:16.346471 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:47:16.346435    2563 generic.go:356] "Generic (PLEG): container finished" podID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerID="67c23b4556a95baf66e5b72fc06da20bd865d95d2bd8cc61a4d1e50b0ba6de4f" exitCode=0
Jul 21 17:47:16.347039 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:47:16.346513    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" event={"ID":"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce","Type":"ContainerDied","Data":"67c23b4556a95baf66e5b72fc06da20bd865d95d2bd8cc61a4d1e50b0ba6de4f"}
Jul 21 17:47:16.348100 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:47:16.348085    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:49:19.954485 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:49:19.954450    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" event={"ID":"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce","Type":"ContainerStarted","Data":"6d3b49c45b4d7d12e52e0182d4fd6cf7e16f8e58f7b671fcde6e6e023a254742"}
Jul 21 17:49:19.954913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:49:19.954516    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:49:19.954913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:49:19.954540    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" event={"ID":"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce","Type":"ContainerStarted","Data":"89bf9caea247d985fbecccbcf3de5999bfb72a608898e836dbc12a225a31a2c6"}
Jul 21 17:49:19.954913 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:49:19.954570    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:49:19.984995 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:49:19.984933    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" podStartSLOduration=1.747893938 podStartE2EDuration="2m33.984916519s" totalImagesPullingTime="2m2.974251272s" totalInitContainerRuntime="29.262771309s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:46:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:49:19.983455103 +0000 UTC m=+1419.541462210" watchObservedRunningTime="2026-07-21 17:49:19.984916519 +0000 UTC m=+1419.542923625"
Jul 21 17:49:25.964814 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:49:25.964759    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:50:40.912073 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:50:40.912047    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:50:40.912970 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:50:40.912952    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:50:55.908238 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:50:55.908206    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:51:03.181221 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:03.181140    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"]
Jul 21 17:51:03.181600 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:03.181452    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="kserve-container" containerID="cri-o://89bf9caea247d985fbecccbcf3de5999bfb72a608898e836dbc12a225a31a2c6" gracePeriod=30
Jul 21 17:51:03.181600 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:03.181473    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="kube-rbac-proxy" containerID="cri-o://6d3b49c45b4d7d12e52e0182d4fd6cf7e16f8e58f7b671fcde6e6e023a254742" gracePeriod=30
Jul 21 17:51:03.399709 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:03.399670    2563 generic.go:356] "Generic (PLEG): container finished" podID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerID="6d3b49c45b4d7d12e52e0182d4fd6cf7e16f8e58f7b671fcde6e6e023a254742" exitCode=2
Jul 21 17:51:03.399894 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:03.399744    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" event={"ID":"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce","Type":"ContainerDied","Data":"6d3b49c45b4d7d12e52e0182d4fd6cf7e16f8e58f7b671fcde6e6e023a254742"}
Jul 21 17:51:04.407328 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.407299    2563 generic.go:356] "Generic (PLEG): container finished" podID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerID="89bf9caea247d985fbecccbcf3de5999bfb72a608898e836dbc12a225a31a2c6" exitCode=0
Jul 21 17:51:04.407652 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.407367    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" event={"ID":"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce","Type":"ContainerDied","Data":"89bf9caea247d985fbecccbcf3de5999bfb72a608898e836dbc12a225a31a2c6"}
Jul 21 17:51:04.527040 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.527015    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:51:04.590454 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.590380    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-rbac-proxy-sar-config\") pod \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") "
Jul 21 17:51:04.590454 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.590441    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kserve-provision-location\") pod \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") "
Jul 21 17:51:04.590657 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.590542    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-cabundle-cert\") pod \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") "
Jul 21 17:51:04.590657 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.590605    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ts24s\" (UniqueName: \"kubernetes.io/projected/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-api-access-ts24s\") pod \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") "
Jul 21 17:51:04.590657 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.590639    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-proxy-tls\") pod \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\" (UID: \"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce\") "
Jul 21 17:51:04.590846 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.590780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" (UID: "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:51:04.590914 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.590894    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" (UID: "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:51:04.590976 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.590961    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:51:04.592858 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.592826    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-api-access-ts24s" (OuterVolumeSpecName: "kube-api-access-ts24s") pod "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" (UID: "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce"). InnerVolumeSpecName "kube-api-access-ts24s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:51:04.592858 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.592836    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" (UID: "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:51:04.595399 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.595378    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" (UID: "65d9cf7c-7229-49ef-8ca9-15e83f98e6ce"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:51:04.691332 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.691304    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:51:04.691332 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.691329    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:51:04.691332 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.691339    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ts24s\" (UniqueName: \"kubernetes.io/projected/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-kube-api-access-ts24s\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:51:04.691546 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:04.691348    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:51:05.415196 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:05.414222    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7" event={"ID":"65d9cf7c-7229-49ef-8ca9-15e83f98e6ce","Type":"ContainerDied","Data":"6cd43b1f5efd82d88d010576d2d781e47d2bf9c54d67507862fd68d56697c20f"}
Jul 21 17:51:05.415196 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:05.414336    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"
Jul 21 17:51:05.415196 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:05.414342    2563 scope.go:117] "RemoveContainer" containerID="6d3b49c45b4d7d12e52e0182d4fd6cf7e16f8e58f7b671fcde6e6e023a254742"
Jul 21 17:51:05.429557 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:05.429533    2563 scope.go:117] "RemoveContainer" containerID="89bf9caea247d985fbecccbcf3de5999bfb72a608898e836dbc12a225a31a2c6"
Jul 21 17:51:05.438604 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:05.438576    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"]
Jul 21 17:51:05.439227 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:05.439207    2563 scope.go:117] "RemoveContainer" containerID="67c23b4556a95baf66e5b72fc06da20bd865d95d2bd8cc61a4d1e50b0ba6de4f"
Jul 21 17:51:05.442831 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:05.442810    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-v2-variants-predictor-5c889d8844-qc6v7"]
Jul 21 17:51:06.908757 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:51:06.908719    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" path="/var/lib/kubelet/pods/65d9cf7c-7229-49ef-8ca9-15e83f98e6ce/volumes"
Jul 21 17:55:40.940240 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:55:40.940211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:55:40.943173 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:55:40.943142    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 17:56:33.191577 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.191538    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"]
Jul 21 17:56:33.192069 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.191908    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="kube-rbac-proxy"
Jul 21 17:56:33.192069 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.191920    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="kube-rbac-proxy"
Jul 21 17:56:33.192069 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.191929    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="storage-initializer"
Jul 21 17:56:33.192069 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.191937    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="storage-initializer"
Jul 21 17:56:33.192069 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.191960    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="kserve-container"
Jul 21 17:56:33.192069 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.191966    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="kserve-container"
Jul 21 17:56:33.192254 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.192082    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="kube-rbac-proxy"
Jul 21 17:56:33.192254 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.192094    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="65d9cf7c-7229-49ef-8ca9-15e83f98e6ce" containerName="kserve-container"
Jul 21 17:56:33.194323 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.194307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.196778 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.196754    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 21 17:56:33.196939 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.196899    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 17:56:33.197106 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.197090    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 17:56:33.197180 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.197163    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 17:56:33.198052 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.198036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 17:56:33.198146 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.198054    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 21 17:56:33.198146 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.198123    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 17:56:33.204604 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.204583    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"]
Jul 21 17:56:33.264836 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.264808    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103c4a75-b504-47d5-bfc4-16809bc23f6a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.264996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.264856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jn7mj\" (UniqueName: \"kubernetes.io/projected/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-api-access-jn7mj\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.264996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.264897    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.264996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.264929    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103c4a75-b504-47d5-bfc4-16809bc23f6a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.264996 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.264951    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.366036 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.366011    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.366168 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.366081    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103c4a75-b504-47d5-bfc4-16809bc23f6a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.366168 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.366132    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jn7mj\" (UniqueName: \"kubernetes.io/projected/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-api-access-jn7mj\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.366258 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.366176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.366258 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.366209    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103c4a75-b504-47d5-bfc4-16809bc23f6a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.366532 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.366511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103c4a75-b504-47d5-bfc4-16809bc23f6a-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.366828 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.366809    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.366884 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.366821    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.368613 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.368597    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103c4a75-b504-47d5-bfc4-16809bc23f6a-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.374346 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.374323    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jn7mj\" (UniqueName: \"kubernetes.io/projected/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-api-access-jn7mj\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.505345 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.505314    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:33.639960 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.639662    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"]
Jul 21 17:56:33.800282 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.800195    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" event={"ID":"103c4a75-b504-47d5-bfc4-16809bc23f6a","Type":"ContainerStarted","Data":"4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d"}
Jul 21 17:56:33.800282 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:33.800257    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" event={"ID":"103c4a75-b504-47d5-bfc4-16809bc23f6a","Type":"ContainerStarted","Data":"94e8d7b05cd6f73245bf027ca8928e77322bdf37cf0d704af71e6155e05c0bac"}
Jul 21 17:56:37.820141 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:37.820109    2563 generic.go:356] "Generic (PLEG): container finished" podID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerID="4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d" exitCode=0
Jul 21 17:56:37.820523 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:37.820180    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" event={"ID":"103c4a75-b504-47d5-bfc4-16809bc23f6a","Type":"ContainerDied","Data":"4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d"}
Jul 21 17:56:37.821388 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:37.821370    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 17:56:44.857688 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:44.857645    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" event={"ID":"103c4a75-b504-47d5-bfc4-16809bc23f6a","Type":"ContainerStarted","Data":"76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9"}
Jul 21 17:56:44.858142 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:44.857716    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" event={"ID":"103c4a75-b504-47d5-bfc4-16809bc23f6a","Type":"ContainerStarted","Data":"d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679"}
Jul 21 17:56:44.858142 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:44.857938    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:44.877501 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:44.877447    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podStartSLOduration=2.198333597 podStartE2EDuration="11.877433233s" totalImagesPullingTime="6.027335252s" totalInitContainerRuntime="3.651764384s" isStatefulPod=false podCreationTimestamp="2026-07-21 17:56:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 17:56:44.876974076 +0000 UTC m=+1864.434981233" watchObservedRunningTime="2026-07-21 17:56:44.877433233 +0000 UTC m=+1864.435440339"
Jul 21 17:56:45.862283 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:45.862253    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:45.863589 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:45.863556    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:56:46.866781 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:46.866746    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:56:51.871410 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:51.871384    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:56:51.871893 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:56:51.871862    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:57:01.872344 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:57:01.872263    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:57:11.872257 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:57:11.872210    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:57:21.872534 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:57:21.872490    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:57:31.872599 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:57:31.872555    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:57:41.872121 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:57:41.872077    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:57:51.871985 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:57:51.871943    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 21 17:58:01.872929 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:01.872900    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:58:04.399276 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:04.399245    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"]
Jul 21 17:58:04.399744 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:04.399540    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container" containerID="cri-o://d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679" gracePeriod=30
Jul 21 17:58:04.399866 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:04.399761    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kube-rbac-proxy" containerID="cri-o://76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9" gracePeriod=30
Jul 21 17:58:05.199051 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:05.199013    2563 generic.go:356] "Generic (PLEG): container finished" podID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerID="76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9" exitCode=2
Jul 21 17:58:05.199225 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:05.199065    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" event={"ID":"103c4a75-b504-47d5-bfc4-16809bc23f6a","Type":"ContainerDied","Data":"76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9"}
Jul 21 17:58:06.867649 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:06.867602    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 21 17:58:09.547910 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.547887    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:58:09.667799 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.667709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-cabundle-cert\") pod \"103c4a75-b504-47d5-bfc4-16809bc23f6a\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") "
Jul 21 17:58:09.667799 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.667773    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103c4a75-b504-47d5-bfc4-16809bc23f6a-proxy-tls\") pod \"103c4a75-b504-47d5-bfc4-16809bc23f6a\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") "
Jul 21 17:58:09.668031 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.667831    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jn7mj\" (UniqueName: \"kubernetes.io/projected/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-api-access-jn7mj\") pod \"103c4a75-b504-47d5-bfc4-16809bc23f6a\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") "
Jul 21 17:58:09.668031 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.667870    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103c4a75-b504-47d5-bfc4-16809bc23f6a-kserve-provision-location\") pod \"103c4a75-b504-47d5-bfc4-16809bc23f6a\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") "
Jul 21 17:58:09.668031 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.667894    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-rbac-proxy-sar-config\") pod \"103c4a75-b504-47d5-bfc4-16809bc23f6a\" (UID: \"103c4a75-b504-47d5-bfc4-16809bc23f6a\") "
Jul 21 17:58:09.668193 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.668089    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "103c4a75-b504-47d5-bfc4-16809bc23f6a" (UID: "103c4a75-b504-47d5-bfc4-16809bc23f6a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:58:09.668251 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.668184    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/103c4a75-b504-47d5-bfc4-16809bc23f6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "103c4a75-b504-47d5-bfc4-16809bc23f6a" (UID: "103c4a75-b504-47d5-bfc4-16809bc23f6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 17:58:09.668292 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.668258    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "103c4a75-b504-47d5-bfc4-16809bc23f6a" (UID: "103c4a75-b504-47d5-bfc4-16809bc23f6a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 17:58:09.670057 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.670037    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/103c4a75-b504-47d5-bfc4-16809bc23f6a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "103c4a75-b504-47d5-bfc4-16809bc23f6a" (UID: "103c4a75-b504-47d5-bfc4-16809bc23f6a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 17:58:09.670057 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.670044    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-api-access-jn7mj" (OuterVolumeSpecName: "kube-api-access-jn7mj") pod "103c4a75-b504-47d5-bfc4-16809bc23f6a" (UID: "103c4a75-b504-47d5-bfc4-16809bc23f6a"). InnerVolumeSpecName "kube-api-access-jn7mj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 17:58:09.768846 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.768812    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/103c4a75-b504-47d5-bfc4-16809bc23f6a-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:58:09.768846 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.768838    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jn7mj\" (UniqueName: \"kubernetes.io/projected/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-api-access-jn7mj\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:58:09.768846 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.768850    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/103c4a75-b504-47d5-bfc4-16809bc23f6a-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:58:09.769038 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.768858    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:58:09.769038 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:09.768867    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/103c4a75-b504-47d5-bfc4-16809bc23f6a-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 17:58:10.225073 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.225040    2563 generic.go:356] "Generic (PLEG): container finished" podID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerID="d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679" exitCode=0
Jul 21 17:58:10.225073 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.225075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" event={"ID":"103c4a75-b504-47d5-bfc4-16809bc23f6a","Type":"ContainerDied","Data":"d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679"}
Jul 21 17:58:10.225301 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.225102    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"
Jul 21 17:58:10.225301 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.225125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g" event={"ID":"103c4a75-b504-47d5-bfc4-16809bc23f6a","Type":"ContainerDied","Data":"94e8d7b05cd6f73245bf027ca8928e77322bdf37cf0d704af71e6155e05c0bac"}
Jul 21 17:58:10.225301 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.225150    2563 scope.go:117] "RemoveContainer" containerID="76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9"
Jul 21 17:58:10.234522 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.234506    2563 scope.go:117] "RemoveContainer" containerID="d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679"
Jul 21 17:58:10.241538 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.241520    2563 scope.go:117] "RemoveContainer" containerID="4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d"
Jul 21 17:58:10.247338 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.247317    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"]
Jul 21 17:58:10.249142 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.249126    2563 scope.go:117] "RemoveContainer" containerID="76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9"
Jul 21 17:58:10.249385 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:58:10.249368    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9\": container with ID starting with 76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9 not found: ID does not exist" containerID="76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9"
Jul 21 17:58:10.249430 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.249393    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9"} err="failed to get container status \"76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9\": rpc error: code = NotFound desc = could not find container \"76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9\": container with ID starting with 76bfdaf5bcda2366a2b3053b97f48cf26bea5456c29b487998cccaf27a7e56b9 not found: ID does not exist"
Jul 21 17:58:10.249430 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.249412    2563 scope.go:117] "RemoveContainer" containerID="d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679"
Jul 21 17:58:10.249650 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:58:10.249631    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679\": container with ID starting with d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679 not found: ID does not exist" containerID="d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679"
Jul 21 17:58:10.249717 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.249658    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679"} err="failed to get container status \"d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679\": rpc error: code = NotFound desc = could not find container \"d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679\": container with ID starting with d1162f9cff691859d312f36ee2c6d034eca5956f4ceded0d99156a174fef8679 not found: ID does not exist"
Jul 21 17:58:10.249717 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.249683    2563 scope.go:117] "RemoveContainer" containerID="4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d"
Jul 21 17:58:10.249961 ip-10-0-129-22 kubenswrapper[2563]: E0721 17:58:10.249947    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d\": container with ID starting with 4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d not found: ID does not exist" containerID="4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d"
Jul 21 17:58:10.250006 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.249963    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d"} err="failed to get container status \"4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d\": rpc error: code = NotFound desc = could not find container \"4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d\": container with ID starting with 4f819b750d9f2789092240e832e3df3217c857f62f659a954682b6ffc56d221d not found: ID does not exist"
Jul 21 17:58:10.253320 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.253298    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-srm7g"]
Jul 21 17:58:10.908534 ip-10-0-129-22 kubenswrapper[2563]: I0721 17:58:10.908503    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" path="/var/lib/kubelet/pods/103c4a75-b504-47d5-bfc4-16809bc23f6a/volumes"
Jul 21 18:00:40.973014 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:00:40.972983    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:00:40.978139 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:00:40.978117    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:04:00.708629 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.708595    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"]
Jul 21 18:04:00.709235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.709112    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kube-rbac-proxy"
Jul 21 18:04:00.709235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.709132    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kube-rbac-proxy"
Jul 21 18:04:00.709235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.709151    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="storage-initializer"
Jul 21 18:04:00.709235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.709161    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="storage-initializer"
Jul 21 18:04:00.709235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.709180    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container"
Jul 21 18:04:00.709235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.709189    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container"
Jul 21 18:04:00.709558 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.709259    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kube-rbac-proxy"
Jul 21 18:04:00.709558 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.709269    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="103c4a75-b504-47d5-bfc4-16809bc23f6a" containerName="kserve-container"
Jul 21 18:04:00.712587 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.712566    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.715277 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.715251    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:04:00.715394 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.715305    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:04:00.715394 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.715309    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:04:00.715486 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.715417    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:04:00.715577 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.715561    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:04:00.716456 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.716436    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 21 18:04:00.716536 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.716458    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 21 18:04:00.724246 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.724219    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"]
Jul 21 18:04:00.771064 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.771024    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.771064 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.771070    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.771295 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.771099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cdtl\" (UniqueName: \"kubernetes.io/projected/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-api-access-9cdtl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.771295 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.771181    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.771295 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.771208    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.872009 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.871972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.872009 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.872014    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.872262 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.872073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.872262 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.872088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.872262 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.872104    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cdtl\" (UniqueName: \"kubernetes.io/projected/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-api-access-9cdtl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.872262 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:04:00.872161    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jul 21 18:04:00.872262 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:04:00.872252    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls podName:38dba9a5-68b6-47d8-9ebc-a5703d5acdf7 nodeName:}" failed. No retries permitted until 2026-07-21 18:04:01.372229121 +0000 UTC m=+2300.930236207 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-z5tls" (UID: "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7") : secret "isvc-paddle-predictor-serving-cert" not found
Jul 21 18:04:00.872558 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.872532    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.872708 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.872688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.872826 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.872776    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:00.882504 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:00.882480    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cdtl\" (UniqueName: \"kubernetes.io/projected/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-api-access-9cdtl\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:01.377314 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:01.377276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:01.379874 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:01.379850    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-z5tls\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:01.622780 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:01.622740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:01.756868 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:01.756830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"]
Jul 21 18:04:01.759166 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:04:01.759136    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38dba9a5_68b6_47d8_9ebc_a5703d5acdf7.slice/crio-0dacdb510448847d31ab45d76855f74d9b0b026f6a8001aa529509504658bab4 WatchSource:0}: Error finding container 0dacdb510448847d31ab45d76855f74d9b0b026f6a8001aa529509504658bab4: Status 404 returned error can't find the container with id 0dacdb510448847d31ab45d76855f74d9b0b026f6a8001aa529509504658bab4
Jul 21 18:04:02.696976 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:02.696942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" event={"ID":"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7","Type":"ContainerStarted","Data":"005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6"}
Jul 21 18:04:02.697152 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:02.697005    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" event={"ID":"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7","Type":"ContainerStarted","Data":"0dacdb510448847d31ab45d76855f74d9b0b026f6a8001aa529509504658bab4"}
Jul 21 18:04:06.715664 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:06.715630    2563 generic.go:356] "Generic (PLEG): container finished" podID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerID="005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6" exitCode=0
Jul 21 18:04:06.716176 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:06.715713    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" event={"ID":"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7","Type":"ContainerDied","Data":"005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6"}
Jul 21 18:04:06.717116 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:06.717101    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 18:04:18.780059 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:18.780023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" event={"ID":"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7","Type":"ContainerStarted","Data":"8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14"}
Jul 21 18:04:18.780526 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:18.780110    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" event={"ID":"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7","Type":"ContainerStarted","Data":"6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565"}
Jul 21 18:04:18.780526 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:18.780389    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:18.780603 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:18.780578    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:18.781550 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:18.781526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 18:04:18.800419 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:18.800262    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podStartSLOduration=2.425761452 podStartE2EDuration="18.800247779s" totalImagesPullingTime="11.568397628s" totalInitContainerRuntime="4.806088699s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:04:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:04:18.799620846 +0000 UTC m=+2318.357627953" watchObservedRunningTime="2026-07-21 18:04:18.800247779 +0000 UTC m=+2318.358254887"
Jul 21 18:04:19.784822 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:19.784763    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 18:04:20.788677 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:20.788635    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 18:04:25.794733 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:25.794696    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:04:25.795256 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:25.795229    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 18:04:35.795421 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:35.795335    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 18:04:45.795618 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:45.795574    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 18:04:55.795116 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:04:55.795078    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 21 18:05:05.796543 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:05.796517    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:05:12.419131 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:12.419085    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"]
Jul 21 18:05:12.419769 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:12.419324    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container" containerID="cri-o://6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565" gracePeriod=30
Jul 21 18:05:12.419769 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:12.419388    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kube-rbac-proxy" containerID="cri-o://8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14" gracePeriod=30
Jul 21 18:05:13.001386 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:13.001354    2563 generic.go:356] "Generic (PLEG): container finished" podID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerID="8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14" exitCode=2
Jul 21 18:05:13.001580 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:13.001430    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" event={"ID":"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7","Type":"ContainerDied","Data":"8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14"}
Jul 21 18:05:15.159989 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.159962    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:05:15.237294 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.237218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-cabundle-cert\") pod \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") "
Jul 21 18:05:15.237294 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.237275    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls\") pod \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") "
Jul 21 18:05:15.237522 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.237321    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-rbac-proxy-sar-config\") pod \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") "
Jul 21 18:05:15.237522 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.237342    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cdtl\" (UniqueName: \"kubernetes.io/projected/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-api-access-9cdtl\") pod \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") "
Jul 21 18:05:15.237522 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.237384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kserve-provision-location\") pod \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\" (UID: \"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7\") "
Jul 21 18:05:15.237680 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.237639    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" (UID: "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:05:15.237873 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.237837    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" (UID: "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:05:15.239743 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.239717    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-api-access-9cdtl" (OuterVolumeSpecName: "kube-api-access-9cdtl") pod "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" (UID: "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7"). InnerVolumeSpecName "kube-api-access-9cdtl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:05:15.239913 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.239895    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" (UID: "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:05:15.247187 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.247161    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" (UID: "38dba9a5-68b6-47d8-9ebc-a5703d5acdf7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:05:15.338143 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.338111    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:05:15.338143 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.338137    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:05:15.338143 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.338146    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:05:15.338143 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.338154    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cdtl\" (UniqueName: \"kubernetes.io/projected/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kube-api-access-9cdtl\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:05:15.338406 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:15.338163    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:05:16.016294 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.016252    2563 generic.go:356] "Generic (PLEG): container finished" podID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerID="6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565" exitCode=0
Jul 21 18:05:16.016470 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.016306    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"
Jul 21 18:05:16.016470 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.016334    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" event={"ID":"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7","Type":"ContainerDied","Data":"6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565"}
Jul 21 18:05:16.016470 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.016398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls" event={"ID":"38dba9a5-68b6-47d8-9ebc-a5703d5acdf7","Type":"ContainerDied","Data":"0dacdb510448847d31ab45d76855f74d9b0b026f6a8001aa529509504658bab4"}
Jul 21 18:05:16.016470 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.016425    2563 scope.go:117] "RemoveContainer" containerID="8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14"
Jul 21 18:05:16.025932 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.025916    2563 scope.go:117] "RemoveContainer" containerID="6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565"
Jul 21 18:05:16.033773 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.033593    2563 scope.go:117] "RemoveContainer" containerID="005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6"
Jul 21 18:05:16.038558 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.038538    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"]
Jul 21 18:05:16.041642 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.041621    2563 scope.go:117] "RemoveContainer" containerID="8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14"
Jul 21 18:05:16.041960 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:05:16.041938    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14\": container with ID starting with 8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14 not found: ID does not exist" containerID="8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14"
Jul 21 18:05:16.042056 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.041973    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14"} err="failed to get container status \"8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14\": rpc error: code = NotFound desc = could not find container \"8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14\": container with ID starting with 8fb93b9b5946327a2804aa85c23780a0cda57f7ea5ad34f77e7b1bc94b8b7e14 not found: ID does not exist"
Jul 21 18:05:16.042056 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.041995    2563 scope.go:117] "RemoveContainer" containerID="6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565"
Jul 21 18:05:16.042242 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:05:16.042224    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565\": container with ID starting with 6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565 not found: ID does not exist" containerID="6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565"
Jul 21 18:05:16.042310 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.042251    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565"} err="failed to get container status \"6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565\": rpc error: code = NotFound desc = could not find container \"6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565\": container with ID starting with 6b09339a638a57893a83125a44195e83881a3ba22454fe16dd60d7e34f0d6565 not found: ID does not exist"
Jul 21 18:05:16.042310 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.042274    2563 scope.go:117] "RemoveContainer" containerID="005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6"
Jul 21 18:05:16.042509 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:05:16.042494    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6\": container with ID starting with 005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6 not found: ID does not exist" containerID="005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6"
Jul 21 18:05:16.042583 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.042515    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6"} err="failed to get container status \"005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6\": rpc error: code = NotFound desc = could not find container \"005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6\": container with ID starting with 005e35b7746c6b472b9afc5bb4de7aa3e8b345334bbdfcbb28e60ffaa8ed65b6 not found: ID does not exist"
Jul 21 18:05:16.042635 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.042581    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-z5tls"]
Jul 21 18:05:16.907674 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:16.907636    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" path="/var/lib/kubelet/pods/38dba9a5-68b6-47d8-9ebc-a5703d5acdf7/volumes"
Jul 21 18:05:41.000511 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:41.000483    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:05:41.006684 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:05:41.006664    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:06:34.445772 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.445734    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"]
Jul 21 18:06:34.446270 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.446167    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kube-rbac-proxy"
Jul 21 18:06:34.446270 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.446181    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kube-rbac-proxy"
Jul 21 18:06:34.446270 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.446201    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container"
Jul 21 18:06:34.446270 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.446220    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container"
Jul 21 18:06:34.446270 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.446228    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="storage-initializer"
Jul 21 18:06:34.446270 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.446234    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="storage-initializer"
Jul 21 18:06:34.446460 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.446301    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kserve-container"
Jul 21 18:06:34.446460 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.446313    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="38dba9a5-68b6-47d8-9ebc-a5703d5acdf7" containerName="kube-rbac-proxy"
Jul 21 18:06:34.448505 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.448485    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.451666 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.451634    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 21 18:06:34.451666 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.451651    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:06:34.451914 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.451708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 21 18:06:34.451914 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.451741    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:06:34.452366 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.452345    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:06:34.452457 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.452406    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:06:34.453003 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.452988    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:06:34.462200 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.462172    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"]
Jul 21 18:06:34.535701 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.535658    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wr5fj\" (UniqueName: \"kubernetes.io/projected/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-api-access-wr5fj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.535701 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.535705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9efb33b-e40d-4239-9eac-4835271e3c0c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.535971 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.535729    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9efb33b-e40d-4239-9eac-4835271e3c0c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.535971 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.535879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.535971 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.535918    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.636411 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.636375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9efb33b-e40d-4239-9eac-4835271e3c0c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.636639 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.636431    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.636639 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.636550    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.636760 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.636669    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wr5fj\" (UniqueName: \"kubernetes.io/projected/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-api-access-wr5fj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.636760 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.636715    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9efb33b-e40d-4239-9eac-4835271e3c0c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.637086 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.637063    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9efb33b-e40d-4239-9eac-4835271e3c0c-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.637231 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.637193    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.637442 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.637260    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.639234 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.639207    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9efb33b-e40d-4239-9eac-4835271e3c0c-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.646359 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.646328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wr5fj\" (UniqueName: \"kubernetes.io/projected/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-api-access-wr5fj\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.759703 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.759652    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:34.901670 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:34.901638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"]
Jul 21 18:06:34.905836 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:06:34.905807    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9efb33b_e40d_4239_9eac_4835271e3c0c.slice/crio-a48222a071092f2db259cb383e7f4dda7f1161a3bde0ae540219005ba9742625 WatchSource:0}: Error finding container a48222a071092f2db259cb383e7f4dda7f1161a3bde0ae540219005ba9742625: Status 404 returned error can't find the container with id a48222a071092f2db259cb383e7f4dda7f1161a3bde0ae540219005ba9742625
Jul 21 18:06:35.352192 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:35.352154    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" event={"ID":"f9efb33b-e40d-4239-9eac-4835271e3c0c","Type":"ContainerStarted","Data":"c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e"}
Jul 21 18:06:35.352373 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:35.352228    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" event={"ID":"f9efb33b-e40d-4239-9eac-4835271e3c0c","Type":"ContainerStarted","Data":"a48222a071092f2db259cb383e7f4dda7f1161a3bde0ae540219005ba9742625"}
Jul 21 18:06:41.381470 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:41.381436    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerID="c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e" exitCode=0
Jul 21 18:06:41.381933 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:41.381511    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" event={"ID":"f9efb33b-e40d-4239-9eac-4835271e3c0c","Type":"ContainerDied","Data":"c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e"}
Jul 21 18:06:42.388371 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:42.388336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" event={"ID":"f9efb33b-e40d-4239-9eac-4835271e3c0c","Type":"ContainerStarted","Data":"4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431"}
Jul 21 18:06:42.388764 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:42.388424    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" event={"ID":"f9efb33b-e40d-4239-9eac-4835271e3c0c","Type":"ContainerStarted","Data":"1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d"}
Jul 21 18:06:42.388764 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:42.388674    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:42.411451 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:42.411382    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podStartSLOduration=2.406133158 podStartE2EDuration="8.411363539s" totalImagesPullingTime="0s" totalInitContainerRuntime="6.005230381s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:06:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:06:42.409225572 +0000 UTC m=+2461.967232703" watchObservedRunningTime="2026-07-21 18:06:42.411363539 +0000 UTC m=+2461.969370646"
Jul 21 18:06:43.392328 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:43.392301    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:43.393425 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:43.393399    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 18:06:44.396411 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:44.396371    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 18:06:49.400559 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:49.400525    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:06:49.401215 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:49.401176    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 18:06:59.401679 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:06:59.401639    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 18:07:09.401311 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:09.401268    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 18:07:19.401178 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:19.401132    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 21 18:07:29.401703 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:29.401670    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:07:36.402621 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:36.402542    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"]
Jul 21 18:07:36.403084 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:36.402878    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container" containerID="cri-o://1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d" gracePeriod=30
Jul 21 18:07:36.403084 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:36.402956    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kube-rbac-proxy" containerID="cri-o://4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431" gracePeriod=30
Jul 21 18:07:36.620590 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:36.620555    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerID="4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431" exitCode=2
Jul 21 18:07:36.620769 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:36.620633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" event={"ID":"f9efb33b-e40d-4239-9eac-4835271e3c0c","Type":"ContainerDied","Data":"4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431"}
Jul 21 18:07:39.359007 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.358981    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:07:39.407017 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.406912    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-rbac-proxy-sar-config\") pod \"f9efb33b-e40d-4239-9eac-4835271e3c0c\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") "
Jul 21 18:07:39.407017 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.406969    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9efb33b-e40d-4239-9eac-4835271e3c0c-proxy-tls\") pod \"f9efb33b-e40d-4239-9eac-4835271e3c0c\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") "
Jul 21 18:07:39.407017 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.406994    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-cabundle-cert\") pod \"f9efb33b-e40d-4239-9eac-4835271e3c0c\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") "
Jul 21 18:07:39.407017 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.407019    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wr5fj\" (UniqueName: \"kubernetes.io/projected/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-api-access-wr5fj\") pod \"f9efb33b-e40d-4239-9eac-4835271e3c0c\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") "
Jul 21 18:07:39.407382 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.407091    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9efb33b-e40d-4239-9eac-4835271e3c0c-kserve-provision-location\") pod \"f9efb33b-e40d-4239-9eac-4835271e3c0c\" (UID: \"f9efb33b-e40d-4239-9eac-4835271e3c0c\") "
Jul 21 18:07:39.407492 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.407469    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "f9efb33b-e40d-4239-9eac-4835271e3c0c" (UID: "f9efb33b-e40d-4239-9eac-4835271e3c0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:07:39.407492 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.407468    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "f9efb33b-e40d-4239-9eac-4835271e3c0c" (UID: "f9efb33b-e40d-4239-9eac-4835271e3c0c"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:07:39.409416 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.409381    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9efb33b-e40d-4239-9eac-4835271e3c0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "f9efb33b-e40d-4239-9eac-4835271e3c0c" (UID: "f9efb33b-e40d-4239-9eac-4835271e3c0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:07:39.409535 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.409468    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-api-access-wr5fj" (OuterVolumeSpecName: "kube-api-access-wr5fj") pod "f9efb33b-e40d-4239-9eac-4835271e3c0c" (UID: "f9efb33b-e40d-4239-9eac-4835271e3c0c"). InnerVolumeSpecName "kube-api-access-wr5fj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:07:39.416636 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.416610    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9efb33b-e40d-4239-9eac-4835271e3c0c-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "f9efb33b-e40d-4239-9eac-4835271e3c0c" (UID: "f9efb33b-e40d-4239-9eac-4835271e3c0c"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:07:39.507738 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.507704    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:07:39.507738 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.507733    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/f9efb33b-e40d-4239-9eac-4835271e3c0c-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:07:39.507738 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.507745    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/f9efb33b-e40d-4239-9eac-4835271e3c0c-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:07:39.507984 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.507754    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wr5fj\" (UniqueName: \"kubernetes.io/projected/f9efb33b-e40d-4239-9eac-4835271e3c0c-kube-api-access-wr5fj\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:07:39.507984 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.507762    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/f9efb33b-e40d-4239-9eac-4835271e3c0c-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:07:39.635970 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.635934    2563 generic.go:356] "Generic (PLEG): container finished" podID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerID="1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d" exitCode=0
Jul 21 18:07:39.636175 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.636004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" event={"ID":"f9efb33b-e40d-4239-9eac-4835271e3c0c","Type":"ContainerDied","Data":"1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d"}
Jul 21 18:07:39.636175 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.636033    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"
Jul 21 18:07:39.636175 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.636096    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh" event={"ID":"f9efb33b-e40d-4239-9eac-4835271e3c0c","Type":"ContainerDied","Data":"a48222a071092f2db259cb383e7f4dda7f1161a3bde0ae540219005ba9742625"}
Jul 21 18:07:39.636175 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.636144    2563 scope.go:117] "RemoveContainer" containerID="4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431"
Jul 21 18:07:39.646537 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.646518    2563 scope.go:117] "RemoveContainer" containerID="1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d"
Jul 21 18:07:39.655062 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.655042    2563 scope.go:117] "RemoveContainer" containerID="c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e"
Jul 21 18:07:39.658743 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.658683    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"]
Jul 21 18:07:39.664864 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.664841    2563 scope.go:117] "RemoveContainer" containerID="4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431"
Jul 21 18:07:39.665184 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:07:39.665160    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431\": container with ID starting with 4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431 not found: ID does not exist" containerID="4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431"
Jul 21 18:07:39.665292 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.665194    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431"} err="failed to get container status \"4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431\": rpc error: code = NotFound desc = could not find container \"4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431\": container with ID starting with 4e6f2058f16b98efa81d80eb41415a69012c10ef22a08c3af8697d3ae93a2431 not found: ID does not exist"
Jul 21 18:07:39.665292 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.665213    2563 scope.go:117] "RemoveContainer" containerID="1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d"
Jul 21 18:07:39.665514 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:07:39.665495    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d\": container with ID starting with 1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d not found: ID does not exist" containerID="1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d"
Jul 21 18:07:39.665582 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.665519    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d"} err="failed to get container status \"1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d\": rpc error: code = NotFound desc = could not find container \"1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d\": container with ID starting with 1d6716f08b3b9dc2017fee93050bba6241d11fff6a9944462a8167582a209e0d not found: ID does not exist"
Jul 21 18:07:39.665582 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.665535    2563 scope.go:117] "RemoveContainer" containerID="c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e"
Jul 21 18:07:39.665759 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:07:39.665741    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e\": container with ID starting with c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e not found: ID does not exist" containerID="c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e"
Jul 21 18:07:39.665839 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.665761    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e"} err="failed to get container status \"c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e\": rpc error: code = NotFound desc = could not find container \"c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e\": container with ID starting with c3dc86dbc44413c7f3cb2ce7d0f7e8228ee8ebb5eecb175d6872c4635f7a2c8e not found: ID does not exist"
Jul 21 18:07:39.666191 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:39.666169    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-vqjvh"]
Jul 21 18:07:40.909609 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:40.909577    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" path="/var/lib/kubelet/pods/f9efb33b-e40d-4239-9eac-4835271e3c0c/volumes"
Jul 21 18:07:41.566035 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566001    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"]
Jul 21 18:07:41.566329 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566317    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="storage-initializer"
Jul 21 18:07:41.566378 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566331    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="storage-initializer"
Jul 21 18:07:41.566378 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566340    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kube-rbac-proxy"
Jul 21 18:07:41.566378 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566346    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kube-rbac-proxy"
Jul 21 18:07:41.566378 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566362    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container"
Jul 21 18:07:41.566378 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566369    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container"
Jul 21 18:07:41.566549 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566426    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kube-rbac-proxy"
Jul 21 18:07:41.566549 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.566435    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9efb33b-e40d-4239-9eac-4835271e3c0c" containerName="kserve-container"
Jul 21 18:07:41.571012 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.570995    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.574086 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.574057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:07:41.574217 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.574057    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:07:41.574217 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.574070    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:07:41.574217 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.574099    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-predictor-serving-cert\""
Jul 21 18:07:41.574439 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.574424    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-kube-rbac-proxy-sar-config\""
Jul 21 18:07:41.575327 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.575313    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:07:41.575504 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.575487    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:07:41.581264 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.581239    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"]
Jul 21 18:07:41.626892 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.626851    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.627096 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.626894    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.627096 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.626969    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtkg2\" (UniqueName: \"kubernetes.io/projected/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-api-access-wtkg2\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.627096 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.627031    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.627096 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.627065    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.728126 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.728071    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.728126 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.728126    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.728409 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.728167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtkg2\" (UniqueName: \"kubernetes.io/projected/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-api-access-wtkg2\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.728409 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.728204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.728409 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.728228    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.728409 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:07:41.728301    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-pmml-predictor-serving-cert: secret "isvc-pmml-predictor-serving-cert" not found
Jul 21 18:07:41.728409 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:07:41.728368    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls podName:76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9 nodeName:}" failed. No retries permitted until 2026-07-21 18:07:42.228345376 +0000 UTC m=+2521.786352461 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls") pod "isvc-pmml-predictor-8674795f59-knxdf" (UID: "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9") : secret "isvc-pmml-predictor-serving-cert" not found
Jul 21 18:07:41.728660 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.728632    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kserve-provision-location\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.728814 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.728767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-cabundle-cert\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.728888 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.728871    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:41.739010 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:41.738989    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtkg2\" (UniqueName: \"kubernetes.io/projected/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-api-access-wtkg2\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:42.232585 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:42.232541    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:42.235121 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:42.235099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls\") pod \"isvc-pmml-predictor-8674795f59-knxdf\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") " pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:42.482002 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:42.481951    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:42.644223 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:42.644195    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"]
Jul 21 18:07:42.646889 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:07:42.646849    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76fe69ca_f90c_4e45_a7fb_71bff2ab0ab9.slice/crio-64289ea612e4a97ad03be7ca35b37845f43dc130131ed49eeb3940550b39b03f WatchSource:0}: Error finding container 64289ea612e4a97ad03be7ca35b37845f43dc130131ed49eeb3940550b39b03f: Status 404 returned error can't find the container with id 64289ea612e4a97ad03be7ca35b37845f43dc130131ed49eeb3940550b39b03f
Jul 21 18:07:43.656735 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:43.656695    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" event={"ID":"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9","Type":"ContainerStarted","Data":"03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236"}
Jul 21 18:07:43.657156 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:43.656764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" event={"ID":"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9","Type":"ContainerStarted","Data":"64289ea612e4a97ad03be7ca35b37845f43dc130131ed49eeb3940550b39b03f"}
Jul 21 18:07:46.670925 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:46.670887    2563 generic.go:356] "Generic (PLEG): container finished" podID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerID="03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236" exitCode=0
Jul 21 18:07:46.671470 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:46.670928    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" event={"ID":"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9","Type":"ContainerDied","Data":"03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236"}
Jul 21 18:07:54.718029 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:54.717992    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" event={"ID":"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9","Type":"ContainerStarted","Data":"0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166"}
Jul 21 18:07:54.718451 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:54.718061    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" event={"ID":"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9","Type":"ContainerStarted","Data":"48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8"}
Jul 21 18:07:54.718451 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:54.718245    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:54.740090 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:54.740036    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podStartSLOduration=2.772203144 podStartE2EDuration="13.740017231s" totalImagesPullingTime="7.148264956s" totalInitContainerRuntime="3.819549131s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:07:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:07:54.737866572 +0000 UTC m=+2534.295873677" watchObservedRunningTime="2026-07-21 18:07:54.740017231 +0000 UTC m=+2534.298024342"
Jul 21 18:07:55.721963 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:55.721932    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:07:55.723419 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:55.723376    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:07:56.727202 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:07:56.727157    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:08:01.731867 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:08:01.731838    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:08:01.732368 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:08:01.732342    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:08:11.733052 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:08:11.733013    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:08:21.732357 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:08:21.732319    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:08:31.732950 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:08:31.732911    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:08:41.733092 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:08:41.733049    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:08:51.732566 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:08:51.732523    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:09:01.732865 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:01.732752    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:09:11.732982 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:11.732945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 21 18:09:21.732881 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:21.732851    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:09:32.613229 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:32.613195    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"]
Jul 21 18:09:32.613806 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:32.613541    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container" containerID="cri-o://48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8" gracePeriod=30
Jul 21 18:09:32.613806 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:32.613553    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kube-rbac-proxy" containerID="cri-o://0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166" gracePeriod=30
Jul 21 18:09:33.145206 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:33.145169    2563 generic.go:356] "Generic (PLEG): container finished" podID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerID="0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166" exitCode=2
Jul 21 18:09:33.145206 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:33.145210    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" event={"ID":"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9","Type":"ContainerDied","Data":"0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166"}
Jul 21 18:09:36.456532 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.456507    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:09:36.555699 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.555660    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-rbac-proxy-sar-config\") pod \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") "
Jul 21 18:09:36.555932 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.555744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kserve-provision-location\") pod \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") "
Jul 21 18:09:36.555932 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.555784    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-cabundle-cert\") pod \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") "
Jul 21 18:09:36.555932 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.555853    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls\") pod \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") "
Jul 21 18:09:36.555932 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.555872    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtkg2\" (UniqueName: \"kubernetes.io/projected/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-api-access-wtkg2\") pod \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\" (UID: \"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9\") "
Jul 21 18:09:36.556166 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.556100    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" (UID: "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:09:36.556166 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.556138    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" (UID: "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:09:36.556166 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.556148    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" (UID: "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:09:36.558139 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.558111    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" (UID: "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:09:36.558219 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.558156    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-api-access-wtkg2" (OuterVolumeSpecName: "kube-api-access-wtkg2") pod "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" (UID: "76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9"). InnerVolumeSpecName "kube-api-access-wtkg2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:09:36.656514 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.656478    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:09:36.656514 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.656507    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:09:36.656514 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.656517    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:09:36.656514 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.656525    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtkg2\" (UniqueName: \"kubernetes.io/projected/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-api-access-wtkg2\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:09:36.656812 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:36.656534    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:09:37.164925 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.164827    2563 generic.go:356] "Generic (PLEG): container finished" podID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerID="48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8" exitCode=0
Jul 21 18:09:37.164925 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.164870    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" event={"ID":"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9","Type":"ContainerDied","Data":"48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8"}
Jul 21 18:09:37.164925 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.164890    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"
Jul 21 18:09:37.164925 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.164921    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf" event={"ID":"76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9","Type":"ContainerDied","Data":"64289ea612e4a97ad03be7ca35b37845f43dc130131ed49eeb3940550b39b03f"}
Jul 21 18:09:37.165243 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.164955    2563 scope.go:117] "RemoveContainer" containerID="0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166"
Jul 21 18:09:37.173817 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.173763    2563 scope.go:117] "RemoveContainer" containerID="48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8"
Jul 21 18:09:37.181593 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.181575    2563 scope.go:117] "RemoveContainer" containerID="03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236"
Jul 21 18:09:37.184618 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.184578    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"]
Jul 21 18:09:37.187882 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.187853    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-predictor-8674795f59-knxdf"]
Jul 21 18:09:37.190123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.190096    2563 scope.go:117] "RemoveContainer" containerID="0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166"
Jul 21 18:09:37.190391 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:09:37.190372    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166\": container with ID starting with 0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166 not found: ID does not exist" containerID="0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166"
Jul 21 18:09:37.190442 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.190400    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166"} err="failed to get container status \"0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166\": rpc error: code = NotFound desc = could not find container \"0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166\": container with ID starting with 0cb3734b5815f756548debd156ae800b5020cf43426cabfedb5abc173b15a166 not found: ID does not exist"
Jul 21 18:09:37.190442 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.190420    2563 scope.go:117] "RemoveContainer" containerID="48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8"
Jul 21 18:09:37.193157 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:09:37.193135    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8\": container with ID starting with 48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8 not found: ID does not exist" containerID="48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8"
Jul 21 18:09:37.193253 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.193163    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8"} err="failed to get container status \"48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8\": rpc error: code = NotFound desc = could not find container \"48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8\": container with ID starting with 48b8f7b119e718c838d4e9a901b37a062e9eabda73cf9c4bcc9f2f71dbae55a8 not found: ID does not exist"
Jul 21 18:09:37.193253 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.193182    2563 scope.go:117] "RemoveContainer" containerID="03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236"
Jul 21 18:09:37.193552 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:09:37.193533    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236\": container with ID starting with 03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236 not found: ID does not exist" containerID="03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236"
Jul 21 18:09:37.193612 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:37.193557    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236"} err="failed to get container status \"03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236\": rpc error: code = NotFound desc = could not find container \"03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236\": container with ID starting with 03f411c5e8002143493da911bafd5ef01b958a2637fe282b06a397ed9407d236 not found: ID does not exist"
Jul 21 18:09:38.909106 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:09:38.909065    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" path="/var/lib/kubelet/pods/76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9/volumes"
Jul 21 18:10:41.027822 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:10:41.027771    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:10:41.034685 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:10:41.034666    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:11:24.129854 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.129819    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"]
Jul 21 18:11:24.130347 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.130178    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kube-rbac-proxy"
Jul 21 18:11:24.130347 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.130189    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kube-rbac-proxy"
Jul 21 18:11:24.130347 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.130202    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container"
Jul 21 18:11:24.130347 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.130208    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container"
Jul 21 18:11:24.130347 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.130215    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="storage-initializer"
Jul 21 18:11:24.130347 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.130221    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="storage-initializer"
Jul 21 18:11:24.130347 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.130276    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kserve-container"
Jul 21 18:11:24.130347 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.130285    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="76fe69ca-f90c-4e45-a7fb-71bff2ab0ab9" containerName="kube-rbac-proxy"
Jul 21 18:11:24.132487 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.132469    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.134931 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.134899    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 21 18:11:24.135175 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.135156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:11:24.135308 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.135255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 21 18:11:24.135384 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.135342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:11:24.136246 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.136228    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:11:24.136364 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.136350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:11:24.136478 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.136463    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:11:24.145728 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.145699    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"]
Jul 21 18:11:24.294811 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.294755    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.295013 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.294843    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.295013 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.294889    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl7qr\" (UniqueName: \"kubernetes.io/projected/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-api-access-pl7qr\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.295013 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.294966    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.295146 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.295029    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.396350 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.396248    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.396350 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.396314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.396572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.396355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.396572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.396387    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.396572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.396433    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl7qr\" (UniqueName: \"kubernetes.io/projected/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-api-access-pl7qr\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.396749 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.396728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.397135 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.397092    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.397135 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.397107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.398998 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.398980    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.405409 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.405390    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl7qr\" (UniqueName: \"kubernetes.io/projected/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-api-access-pl7qr\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.443356 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.443320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:24.582202 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.582115    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"]
Jul 21 18:11:24.584568 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:11:24.584539    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88bb48a3_7cb2_429d_bf19_dd5c8ef74ab8.slice/crio-875f1e142bef9700eb00531b963f734813bc4756eb4d78716a8a94b12dad699b WatchSource:0}: Error finding container 875f1e142bef9700eb00531b963f734813bc4756eb4d78716a8a94b12dad699b: Status 404 returned error can't find the container with id 875f1e142bef9700eb00531b963f734813bc4756eb4d78716a8a94b12dad699b
Jul 21 18:11:24.625930 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:24.625890    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" event={"ID":"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8","Type":"ContainerStarted","Data":"875f1e142bef9700eb00531b963f734813bc4756eb4d78716a8a94b12dad699b"}
Jul 21 18:11:25.634095 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:25.634056    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" event={"ID":"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8","Type":"ContainerStarted","Data":"979f7f61560138c08a071563905a436fe76f600ebdb8da4631aee9a6a6dfdcfa"}
Jul 21 18:11:28.648123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:28.648030    2563 generic.go:356] "Generic (PLEG): container finished" podID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerID="979f7f61560138c08a071563905a436fe76f600ebdb8da4631aee9a6a6dfdcfa" exitCode=0
Jul 21 18:11:28.648123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:28.648095    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" event={"ID":"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8","Type":"ContainerDied","Data":"979f7f61560138c08a071563905a436fe76f600ebdb8da4631aee9a6a6dfdcfa"}
Jul 21 18:11:29.654389 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:29.654350    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" event={"ID":"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8","Type":"ContainerStarted","Data":"213c074f5a95f758b454d65bde67defda60c7874461e23e6d038e8213825d581"}
Jul 21 18:11:29.654913 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:29.654417    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" event={"ID":"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8","Type":"ContainerStarted","Data":"0d7c44dc6eb12e5a9fd1b28117a5267a3a096f6c119e12496384a112eee87088"}
Jul 21 18:11:29.654913 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:29.654658    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:29.675601 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:29.675554    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podStartSLOduration=2.071826896 podStartE2EDuration="5.675539174s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.603712278s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:11:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:11:29.673889582 +0000 UTC m=+2749.231896714" watchObservedRunningTime="2026-07-21 18:11:29.675539174 +0000 UTC m=+2749.233546281"
Jul 21 18:11:30.658385 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:30.658341    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:30.659706 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:30.659681    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:11:31.662249 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:31.662212    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:11:36.666807 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:36.666761    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:11:36.667352 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:36.667320    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:11:46.668166 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:46.668125    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:11:56.668254 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:11:56.668209    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:12:06.667546 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:06.667455    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:12:16.667428 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:16.667387    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:12:26.668070 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:26.668027    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:12:36.667307 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:36.667270    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:12:46.667897 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:46.667866    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:12:55.405733 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:55.405699    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"]
Jul 21 18:12:55.406212 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:55.406062    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kube-rbac-proxy" containerID="cri-o://213c074f5a95f758b454d65bde67defda60c7874461e23e6d038e8213825d581" gracePeriod=30
Jul 21 18:12:55.406212 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:55.406093    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" containerID="cri-o://0d7c44dc6eb12e5a9fd1b28117a5267a3a096f6c119e12496384a112eee87088" gracePeriod=30
Jul 21 18:12:56.017947 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:56.017914    2563 generic.go:356] "Generic (PLEG): container finished" podID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerID="213c074f5a95f758b454d65bde67defda60c7874461e23e6d038e8213825d581" exitCode=2
Jul 21 18:12:56.018123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:56.017987    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" event={"ID":"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8","Type":"ContainerDied","Data":"213c074f5a95f758b454d65bde67defda60c7874461e23e6d038e8213825d581"}
Jul 21 18:12:56.663126 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:56.663075    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 21 18:12:56.667816 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:56.667777    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 21 18:12:59.036059 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.036026    2563 generic.go:356] "Generic (PLEG): container finished" podID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerID="0d7c44dc6eb12e5a9fd1b28117a5267a3a096f6c119e12496384a112eee87088" exitCode=0
Jul 21 18:12:59.036514 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.036112    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" event={"ID":"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8","Type":"ContainerDied","Data":"0d7c44dc6eb12e5a9fd1b28117a5267a3a096f6c119e12496384a112eee87088"}
Jul 21 18:12:59.054725 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.054703    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:12:59.200868 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.200752    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl7qr\" (UniqueName: \"kubernetes.io/projected/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-api-access-pl7qr\") pod \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") "
Jul 21 18:12:59.200868 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.200836    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-rbac-proxy-sar-config\") pod \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") "
Jul 21 18:12:59.201103 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.200877    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-cabundle-cert\") pod \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") "
Jul 21 18:12:59.201103 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.200917    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-proxy-tls\") pod \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") "
Jul 21 18:12:59.201103 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.200983    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kserve-provision-location\") pod \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\" (UID: \"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8\") "
Jul 21 18:12:59.201256 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.201228    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" (UID: "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:12:59.201295 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.201234    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" (UID: "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:12:59.201295 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.201279    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" (UID: "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:12:59.202966 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.202945    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-api-access-pl7qr" (OuterVolumeSpecName: "kube-api-access-pl7qr") pod "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" (UID: "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8"). InnerVolumeSpecName "kube-api-access-pl7qr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:12:59.203063 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.203033    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" (UID: "88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:12:59.301907 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.301874    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:12:59.301907 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.301900    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:12:59.301907 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.301911    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:12:59.302122 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.301919    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:12:59.302122 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:12:59.301928    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl7qr\" (UniqueName: \"kubernetes.io/projected/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8-kube-api-access-pl7qr\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:13:00.042521 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:13:00.042493    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"
Jul 21 18:13:00.042521 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:13:00.042517    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks" event={"ID":"88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8","Type":"ContainerDied","Data":"875f1e142bef9700eb00531b963f734813bc4756eb4d78716a8a94b12dad699b"}
Jul 21 18:13:00.043030 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:13:00.042583    2563 scope.go:117] "RemoveContainer" containerID="213c074f5a95f758b454d65bde67defda60c7874461e23e6d038e8213825d581"
Jul 21 18:13:00.051535 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:13:00.051517    2563 scope.go:117] "RemoveContainer" containerID="0d7c44dc6eb12e5a9fd1b28117a5267a3a096f6c119e12496384a112eee87088"
Jul 21 18:13:00.059264 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:13:00.059244    2563 scope.go:117] "RemoveContainer" containerID="979f7f61560138c08a071563905a436fe76f600ebdb8da4631aee9a6a6dfdcfa"
Jul 21 18:13:00.064327 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:13:00.064306    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"]
Jul 21 18:13:00.070001 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:13:00.069978    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g5nks"]
Jul 21 18:13:00.907743 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:13:00.907710    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" path="/var/lib/kubelet/pods/88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8/volumes"
Jul 21 18:15:17.300830 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.300734    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"]
Jul 21 18:15:17.301210 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.301126    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container"
Jul 21 18:15:17.301210 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.301140    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container"
Jul 21 18:15:17.301210 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.301162    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kube-rbac-proxy"
Jul 21 18:15:17.301210 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.301168    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kube-rbac-proxy"
Jul 21 18:15:17.301210 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.301178    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="storage-initializer"
Jul 21 18:15:17.301210 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.301183    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="storage-initializer"
Jul 21 18:15:17.301400 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.301226    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kserve-container"
Jul 21 18:15:17.301400 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.301237    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="88bb48a3-7cb2-429d-bf19-dd5c8ef74ab8" containerName="kube-rbac-proxy"
Jul 21 18:15:17.304549 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.304531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.307343 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.307319    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:15:17.307546 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.307381    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:15:17.307546 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.307393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:15:17.307546 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.307320    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 21 18:15:17.308403 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.308379    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:15:17.308551 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.308462    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 21 18:15:17.308734 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.308718    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:15:17.315468 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.315419    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"]
Jul 21 18:15:17.373488 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.373460    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.373628 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.373517    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.373628 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.373543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2wl25\" (UniqueName: \"kubernetes.io/projected/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-api-access-2wl25\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.373628 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.373587    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b217052-5271-40f7-b2f0-e033c33c0f75-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.373762 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.373635    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b217052-5271-40f7-b2f0-e033c33c0f75-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.474955 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.474910    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b217052-5271-40f7-b2f0-e033c33c0f75-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.475122 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.474978    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.475122 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.475055    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.475122 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.475073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2wl25\" (UniqueName: \"kubernetes.io/projected/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-api-access-2wl25\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.475122 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.475102    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b217052-5271-40f7-b2f0-e033c33c0f75-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.475382 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.475364    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b217052-5271-40f7-b2f0-e033c33c0f75-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.475693 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.475671    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.475825 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.475675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.477607 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.477581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b217052-5271-40f7-b2f0-e033c33c0f75-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.483493 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.483465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2wl25\" (UniqueName: \"kubernetes.io/projected/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-api-access-2wl25\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-p26d6\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.617222 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.617145    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:17.749633 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:17.749606    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"]
Jul 21 18:15:17.752238 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:15:17.752211    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b217052_5271_40f7_b2f0_e033c33c0f75.slice/crio-dfa79c3b2a7ef60e7c56cac0559f0354fdbb74808e4bd6119f4dd8101f13d94c WatchSource:0}: Error finding container dfa79c3b2a7ef60e7c56cac0559f0354fdbb74808e4bd6119f4dd8101f13d94c: Status 404 returned error can't find the container with id dfa79c3b2a7ef60e7c56cac0559f0354fdbb74808e4bd6119f4dd8101f13d94c
Jul 21 18:15:18.620374 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:18.620338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" event={"ID":"9b217052-5271-40f7-b2f0-e033c33c0f75","Type":"ContainerStarted","Data":"8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d"}
Jul 21 18:15:18.620723 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:18.620423    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" event={"ID":"9b217052-5271-40f7-b2f0-e033c33c0f75","Type":"ContainerStarted","Data":"dfa79c3b2a7ef60e7c56cac0559f0354fdbb74808e4bd6119f4dd8101f13d94c"}
Jul 21 18:15:21.633486 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:21.633453    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerID="8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d" exitCode=0
Jul 21 18:15:21.633486 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:21.633490    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" event={"ID":"9b217052-5271-40f7-b2f0-e033c33c0f75","Type":"ContainerDied","Data":"8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d"}
Jul 21 18:15:21.634670 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:21.634653    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 18:15:42.938245 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:42.938209    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:15:42.938852 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:42.938372    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:15:43.756058 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:43.756025    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" event={"ID":"9b217052-5271-40f7-b2f0-e033c33c0f75","Type":"ContainerStarted","Data":"7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785"}
Jul 21 18:15:43.756235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:43.756086    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" event={"ID":"9b217052-5271-40f7-b2f0-e033c33c0f75","Type":"ContainerStarted","Data":"3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9"}
Jul 21 18:15:43.756406 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:43.756387    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:43.756474 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:43.756446    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:43.757667 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:43.757640    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:15:43.776322 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:43.776268    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podStartSLOduration=1.567291021 podStartE2EDuration="26.776252671s" totalImagesPullingTime="21.40912722s" totalInitContainerRuntime="3.79983443s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:15:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:15:43.775473241 +0000 UTC m=+3003.333480367" watchObservedRunningTime="2026-07-21 18:15:43.776252671 +0000 UTC m=+3003.334259772"
Jul 21 18:15:44.760626 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:44.760577    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:15:49.765379 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:49.765351    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:15:49.765867 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:49.765838    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:15:59.766992 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:15:59.766945    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:16:09.765863 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:16:09.765820    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:16:19.766106 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:16:19.766063    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:16:29.766062 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:16:29.766017    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:16:39.766301 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:16:39.766212    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:16:49.766477 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:16:49.766438    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:16:59.767002 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:16:59.766972    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:17:07.439646 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:07.439612    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"]
Jul 21 18:17:07.440157 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:07.439918    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" containerID="cri-o://3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9" gracePeriod=30
Jul 21 18:17:07.440157 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:07.440109    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kube-rbac-proxy" containerID="cri-o://7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785" gracePeriod=30
Jul 21 18:17:08.116456 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:08.116425    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerID="7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785" exitCode=2
Jul 21 18:17:08.116629 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:08.116503    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" event={"ID":"9b217052-5271-40f7-b2f0-e033c33c0f75","Type":"ContainerDied","Data":"7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785"}
Jul 21 18:17:09.760983 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:09.760934    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.34:8643/healthz\": dial tcp 10.133.0.34:8643: connect: connection refused"
Jul 21 18:17:09.766277 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:09.766250    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 21 18:17:11.884100 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.884076    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:17:11.988627 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.988549    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b217052-5271-40f7-b2f0-e033c33c0f75-proxy-tls\") pod \"9b217052-5271-40f7-b2f0-e033c33c0f75\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") "
Jul 21 18:17:11.988777 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.988629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-rbac-proxy-sar-config\") pod \"9b217052-5271-40f7-b2f0-e033c33c0f75\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") "
Jul 21 18:17:11.988777 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.988664    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2wl25\" (UniqueName: \"kubernetes.io/projected/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-api-access-2wl25\") pod \"9b217052-5271-40f7-b2f0-e033c33c0f75\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") "
Jul 21 18:17:11.988777 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.988686    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-cabundle-cert\") pod \"9b217052-5271-40f7-b2f0-e033c33c0f75\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") "
Jul 21 18:17:11.988980 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.988819    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b217052-5271-40f7-b2f0-e033c33c0f75-kserve-provision-location\") pod \"9b217052-5271-40f7-b2f0-e033c33c0f75\" (UID: \"9b217052-5271-40f7-b2f0-e033c33c0f75\") "
Jul 21 18:17:11.989083 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.989052    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9b217052-5271-40f7-b2f0-e033c33c0f75" (UID: "9b217052-5271-40f7-b2f0-e033c33c0f75"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:17:11.989083 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.989074    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b217052-5271-40f7-b2f0-e033c33c0f75-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9b217052-5271-40f7-b2f0-e033c33c0f75" (UID: "9b217052-5271-40f7-b2f0-e033c33c0f75"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:17:11.989189 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.989099    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9b217052-5271-40f7-b2f0-e033c33c0f75" (UID: "9b217052-5271-40f7-b2f0-e033c33c0f75"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:17:11.990721 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.990698    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b217052-5271-40f7-b2f0-e033c33c0f75-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9b217052-5271-40f7-b2f0-e033c33c0f75" (UID: "9b217052-5271-40f7-b2f0-e033c33c0f75"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:17:11.990866 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:11.990776    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-api-access-2wl25" (OuterVolumeSpecName: "kube-api-access-2wl25") pod "9b217052-5271-40f7-b2f0-e033c33c0f75" (UID: "9b217052-5271-40f7-b2f0-e033c33c0f75"). InnerVolumeSpecName "kube-api-access-2wl25". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:17:12.090069 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.090043    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9b217052-5271-40f7-b2f0-e033c33c0f75-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:17:12.090069 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.090069    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:17:12.090255 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.090079    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2wl25\" (UniqueName: \"kubernetes.io/projected/9b217052-5271-40f7-b2f0-e033c33c0f75-kube-api-access-2wl25\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:17:12.090255 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.090089    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9b217052-5271-40f7-b2f0-e033c33c0f75-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:17:12.090255 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.090097    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9b217052-5271-40f7-b2f0-e033c33c0f75-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:17:12.138936 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.138904    2563 generic.go:356] "Generic (PLEG): container finished" podID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerID="3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9" exitCode=0
Jul 21 18:17:12.139087 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.138948    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" event={"ID":"9b217052-5271-40f7-b2f0-e033c33c0f75","Type":"ContainerDied","Data":"3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9"}
Jul 21 18:17:12.139087 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.138965    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"
Jul 21 18:17:12.139087 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.139011    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6" event={"ID":"9b217052-5271-40f7-b2f0-e033c33c0f75","Type":"ContainerDied","Data":"dfa79c3b2a7ef60e7c56cac0559f0354fdbb74808e4bd6119f4dd8101f13d94c"}
Jul 21 18:17:12.139087 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.139045    2563 scope.go:117] "RemoveContainer" containerID="7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785"
Jul 21 18:17:12.148493 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.148477    2563 scope.go:117] "RemoveContainer" containerID="3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9"
Jul 21 18:17:12.156963 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.156950    2563 scope.go:117] "RemoveContainer" containerID="8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d"
Jul 21 18:17:12.161710 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.161688    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"]
Jul 21 18:17:12.165323 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.165297    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-p26d6"]
Jul 21 18:17:12.165772 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.165756    2563 scope.go:117] "RemoveContainer" containerID="7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785"
Jul 21 18:17:12.166080 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:17:12.166063    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785\": container with ID starting with 7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785 not found: ID does not exist" containerID="7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785"
Jul 21 18:17:12.166147 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.166093    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785"} err="failed to get container status \"7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785\": rpc error: code = NotFound desc = could not find container \"7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785\": container with ID starting with 7efc7811291ed14765cb2d204cc3dde0db2bb7c645477b5d73d4a254ffa88785 not found: ID does not exist"
Jul 21 18:17:12.166147 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.166116    2563 scope.go:117] "RemoveContainer" containerID="3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9"
Jul 21 18:17:12.166361 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:17:12.166345    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9\": container with ID starting with 3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9 not found: ID does not exist" containerID="3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9"
Jul 21 18:17:12.166404 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.166368    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9"} err="failed to get container status \"3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9\": rpc error: code = NotFound desc = could not find container \"3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9\": container with ID starting with 3e25354f89bbd134b93fcf1868bde979c1bf1deaf879c06e408cce32e793a7f9 not found: ID does not exist"
Jul 21 18:17:12.166404 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.166384    2563 scope.go:117] "RemoveContainer" containerID="8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d"
Jul 21 18:17:12.166588 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:17:12.166573    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d\": container with ID starting with 8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d not found: ID does not exist" containerID="8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d"
Jul 21 18:17:12.166631 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.166591    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d"} err="failed to get container status \"8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d\": rpc error: code = NotFound desc = could not find container \"8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d\": container with ID starting with 8c2d15356b641986d9b2e0a2dc8c24e31c727375e82d53ea28cbcbd75271295d not found: ID does not exist"
Jul 21 18:17:12.619669 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.619635    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"]
Jul 21 18:17:12.619997 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.619984    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kube-rbac-proxy"
Jul 21 18:17:12.620079 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.619999    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kube-rbac-proxy"
Jul 21 18:17:12.620079 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.620014    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="storage-initializer"
Jul 21 18:17:12.620079 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.620019    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="storage-initializer"
Jul 21 18:17:12.620079 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.620029    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container"
Jul 21 18:17:12.620079 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.620035    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container"
Jul 21 18:17:12.620235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.620082    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kserve-container"
Jul 21 18:17:12.620235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.620092    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" containerName="kube-rbac-proxy"
Jul 21 18:17:12.623219 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.623199    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.625734 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.625713    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:17:12.626175 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.626156    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:17:12.626175 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.626164    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 21 18:17:12.626358 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.626174    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:17:12.626358 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.626160    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 21 18:17:12.626358 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.626230    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:17:12.626358 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.626170    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:17:12.636686 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.636663    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"]
Jul 21 18:17:12.695313 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.695275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.695496 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.695318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e3e147-43ec-48d8-b8b2-61d307983a80-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.695496 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.695376    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.695496 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.695414    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9m5dn\" (UniqueName: \"kubernetes.io/projected/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-api-access-9m5dn\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.695683 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.695526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.796474 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.796414    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9m5dn\" (UniqueName: \"kubernetes.io/projected/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-api-access-9m5dn\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.796823 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.796528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.796823 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.796587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.796823 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.796625    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e3e147-43ec-48d8-b8b2-61d307983a80-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.796823 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:17:12.796664    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 21 18:17:12.796823 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:17:12.796742    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls podName:15e3e147-43ec-48d8-b8b2-61d307983a80 nodeName:}" failed. No retries permitted until 2026-07-21 18:17:13.296720554 +0000 UTC m=+3092.854727641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-kgzlx" (UID: "15e3e147-43ec-48d8-b8b2-61d307983a80") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 21 18:17:12.796823 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.796675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.797179 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.797093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e3e147-43ec-48d8-b8b2-61d307983a80-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.797332 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.797313    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.797401 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.797386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.806655 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.806635    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9m5dn\" (UniqueName: \"kubernetes.io/projected/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-api-access-9m5dn\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:12.908900 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:12.908817    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b217052-5271-40f7-b2f0-e033c33c0f75" path="/var/lib/kubelet/pods/9b217052-5271-40f7-b2f0-e033c33c0f75/volumes"
Jul 21 18:17:13.301274 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:13.301235    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:13.303781 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:13.303755    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-kgzlx\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:13.540825 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:13.540748    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:13.682040 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:13.681997    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"]
Jul 21 18:17:13.685650 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:17:13.685625    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod15e3e147_43ec_48d8_b8b2_61d307983a80.slice/crio-45e32a673841820ff1873be78a79b3ca0c7b55ab0badbf85608b5ba675341bac WatchSource:0}: Error finding container 45e32a673841820ff1873be78a79b3ca0c7b55ab0badbf85608b5ba675341bac: Status 404 returned error can't find the container with id 45e32a673841820ff1873be78a79b3ca0c7b55ab0badbf85608b5ba675341bac
Jul 21 18:17:14.152293 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:14.152246    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" event={"ID":"15e3e147-43ec-48d8-b8b2-61d307983a80","Type":"ContainerStarted","Data":"bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b"}
Jul 21 18:17:14.152703 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:14.152328    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" event={"ID":"15e3e147-43ec-48d8-b8b2-61d307983a80","Type":"ContainerStarted","Data":"45e32a673841820ff1873be78a79b3ca0c7b55ab0badbf85608b5ba675341bac"}
Jul 21 18:17:18.170923 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:18.170891    2563 generic.go:356] "Generic (PLEG): container finished" podID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerID="bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b" exitCode=0
Jul 21 18:17:18.171358 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:18.170961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" event={"ID":"15e3e147-43ec-48d8-b8b2-61d307983a80","Type":"ContainerDied","Data":"bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b"}
Jul 21 18:17:19.178524 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:19.178482    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" event={"ID":"15e3e147-43ec-48d8-b8b2-61d307983a80","Type":"ContainerStarted","Data":"2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db"}
Jul 21 18:17:19.179044 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:19.178597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" event={"ID":"15e3e147-43ec-48d8-b8b2-61d307983a80","Type":"ContainerStarted","Data":"42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c"}
Jul 21 18:17:19.179044 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:19.178965    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:19.179177 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:19.179113    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:19.180145 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:19.180119    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:17:19.201398 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:19.201348    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podStartSLOduration=3.568923247 podStartE2EDuration="7.201334171s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.632410924s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:17:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:17:19.198648457 +0000 UTC m=+3098.756655563" watchObservedRunningTime="2026-07-21 18:17:19.201334171 +0000 UTC m=+3098.759341277"
Jul 21 18:17:20.184057 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:20.184021    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:17:25.188939 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:25.188911    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:17:25.189475 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:25.189444    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:17:35.190053 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:35.190011    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:17:45.189638 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:45.189594    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:17:55.190368 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:17:55.190321    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:18:05.189497 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:05.189409    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:18:15.189567 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:15.189520    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:18:25.189767 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:25.189723    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:18:25.904114 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:25.904066    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:18:35.904984 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:35.904949    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:18:42.731044 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:42.731001    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"]
Jul 21 18:18:42.731521 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:42.731238    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" containerID="cri-o://42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c" gracePeriod=30
Jul 21 18:18:42.731521 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:42.731282    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kube-rbac-proxy" containerID="cri-o://2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db" gracePeriod=30
Jul 21 18:18:43.548035 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:43.548001    2563 generic.go:356] "Generic (PLEG): container finished" podID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerID="2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db" exitCode=2
Jul 21 18:18:43.548222 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:43.548083    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" event={"ID":"15e3e147-43ec-48d8-b8b2-61d307983a80","Type":"ContainerDied","Data":"2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db"}
Jul 21 18:18:45.184746 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:45.184699    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 21 18:18:45.904145 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:45.904100    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 21 18:18:47.380221 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.380193    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:18:47.522599 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.522568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls\") pod \"15e3e147-43ec-48d8-b8b2-61d307983a80\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") "
Jul 21 18:18:47.522767 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.522608    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e3e147-43ec-48d8-b8b2-61d307983a80-kserve-provision-location\") pod \"15e3e147-43ec-48d8-b8b2-61d307983a80\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") "
Jul 21 18:18:47.522767 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.522636    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9m5dn\" (UniqueName: \"kubernetes.io/projected/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-api-access-9m5dn\") pod \"15e3e147-43ec-48d8-b8b2-61d307983a80\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") "
Jul 21 18:18:47.522767 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.522668    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-rbac-proxy-sar-config\") pod \"15e3e147-43ec-48d8-b8b2-61d307983a80\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") "
Jul 21 18:18:47.522767 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.522733    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-cabundle-cert\") pod \"15e3e147-43ec-48d8-b8b2-61d307983a80\" (UID: \"15e3e147-43ec-48d8-b8b2-61d307983a80\") "
Jul 21 18:18:47.523001 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.522979    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/15e3e147-43ec-48d8-b8b2-61d307983a80-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "15e3e147-43ec-48d8-b8b2-61d307983a80" (UID: "15e3e147-43ec-48d8-b8b2-61d307983a80"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:18:47.523147 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.523120    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "15e3e147-43ec-48d8-b8b2-61d307983a80" (UID: "15e3e147-43ec-48d8-b8b2-61d307983a80"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:18:47.523201 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.523122    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "15e3e147-43ec-48d8-b8b2-61d307983a80" (UID: "15e3e147-43ec-48d8-b8b2-61d307983a80"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:18:47.524987 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.524963    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "15e3e147-43ec-48d8-b8b2-61d307983a80" (UID: "15e3e147-43ec-48d8-b8b2-61d307983a80"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:18:47.525093 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.524988    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-api-access-9m5dn" (OuterVolumeSpecName: "kube-api-access-9m5dn") pod "15e3e147-43ec-48d8-b8b2-61d307983a80" (UID: "15e3e147-43ec-48d8-b8b2-61d307983a80"). InnerVolumeSpecName "kube-api-access-9m5dn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:18:47.565518 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.565488    2563 generic.go:356] "Generic (PLEG): container finished" podID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerID="42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c" exitCode=0
Jul 21 18:18:47.565657 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.565545    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"
Jul 21 18:18:47.565657 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.565577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" event={"ID":"15e3e147-43ec-48d8-b8b2-61d307983a80","Type":"ContainerDied","Data":"42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c"}
Jul 21 18:18:47.565657 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.565646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx" event={"ID":"15e3e147-43ec-48d8-b8b2-61d307983a80","Type":"ContainerDied","Data":"45e32a673841820ff1873be78a79b3ca0c7b55ab0badbf85608b5ba675341bac"}
Jul 21 18:18:47.565765 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.565674    2563 scope.go:117] "RemoveContainer" containerID="2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db"
Jul 21 18:18:47.578457 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.578439    2563 scope.go:117] "RemoveContainer" containerID="42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c"
Jul 21 18:18:47.587244 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.587227    2563 scope.go:117] "RemoveContainer" containerID="bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b"
Jul 21 18:18:47.587899 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.587875    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"]
Jul 21 18:18:47.590260 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.590237    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-kgzlx"]
Jul 21 18:18:47.594992 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.594975    2563 scope.go:117] "RemoveContainer" containerID="2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db"
Jul 21 18:18:47.595270 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:18:47.595252    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db\": container with ID starting with 2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db not found: ID does not exist" containerID="2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db"
Jul 21 18:18:47.595322 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.595279    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db"} err="failed to get container status \"2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db\": rpc error: code = NotFound desc = could not find container \"2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db\": container with ID starting with 2326ea1da68fb46321242f68a1564144f8e8987524907e5e503e54424acc17db not found: ID does not exist"
Jul 21 18:18:47.595322 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.595300    2563 scope.go:117] "RemoveContainer" containerID="42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c"
Jul 21 18:18:47.595559 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:18:47.595540    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c\": container with ID starting with 42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c not found: ID does not exist" containerID="42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c"
Jul 21 18:18:47.595606 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.595565    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c"} err="failed to get container status \"42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c\": rpc error: code = NotFound desc = could not find container \"42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c\": container with ID starting with 42728756bed193f3b53b6f8459c1f24ddf6fb19c1b6d0e58d1d01323d52f027c not found: ID does not exist"
Jul 21 18:18:47.595606 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.595582    2563 scope.go:117] "RemoveContainer" containerID="bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b"
Jul 21 18:18:47.595835 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:18:47.595815    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b\": container with ID starting with bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b not found: ID does not exist" containerID="bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b"
Jul 21 18:18:47.595899 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.595844    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b"} err="failed to get container status \"bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b\": rpc error: code = NotFound desc = could not find container \"bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b\": container with ID starting with bbbb9e7182f0462eeaa85bb105673405fe68ff6f3910c180def0c6adad73952b not found: ID does not exist"
Jul 21 18:18:47.624244 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.624217    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:18:47.624244 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.624244    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/15e3e147-43ec-48d8-b8b2-61d307983a80-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:18:47.624382 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.624254    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/15e3e147-43ec-48d8-b8b2-61d307983a80-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:18:47.624382 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.624263    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9m5dn\" (UniqueName: \"kubernetes.io/projected/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-api-access-9m5dn\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:18:47.624382 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.624292    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/15e3e147-43ec-48d8-b8b2-61d307983a80-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:18:47.895077 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.894988    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"]
Jul 21 18:18:47.895334 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.895322    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="storage-initializer"
Jul 21 18:18:47.895384 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.895336    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="storage-initializer"
Jul 21 18:18:47.895384 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.895348    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container"
Jul 21 18:18:47.895384 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.895354    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container"
Jul 21 18:18:47.895384 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.895366    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kube-rbac-proxy"
Jul 21 18:18:47.895384 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.895372    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kube-rbac-proxy"
Jul 21 18:18:47.895563 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.895442    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kserve-container"
Jul 21 18:18:47.895563 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.895453    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" containerName="kube-rbac-proxy"
Jul 21 18:18:47.898677 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.898658    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:47.901314 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.901287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:18:47.901478 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.901318    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:18:47.901478 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.901294    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 21 18:18:47.901478 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.901288    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 21 18:18:47.901478 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.901342    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:18:47.901669 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.901565    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:18:47.901712 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.901706    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:18:47.908742 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:47.908720    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"]
Jul 21 18:18:48.026978 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.026942    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66de434f-8eae-4260-95b1-11ddc5cd922b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.027215 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.027189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.027335 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.027314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66de434f-8eae-4260-95b1-11ddc5cd922b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.027693 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.027667    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8pv7\" (UniqueName: \"kubernetes.io/projected/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-api-access-q8pv7\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.027757 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.027724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.128371 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.128334    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66de434f-8eae-4260-95b1-11ddc5cd922b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.128562 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.128398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8pv7\" (UniqueName: \"kubernetes.io/projected/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-api-access-q8pv7\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.128562 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.128424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.128562 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.128478    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66de434f-8eae-4260-95b1-11ddc5cd922b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.128562 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.128532    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.128821 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.128757    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66de434f-8eae-4260-95b1-11ddc5cd922b-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.129147 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.129125    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.129147 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.129137    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.131160 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.131129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66de434f-8eae-4260-95b1-11ddc5cd922b-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.136911 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.136886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8pv7\" (UniqueName: \"kubernetes.io/projected/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-api-access-q8pv7\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.209332 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.209241    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:48.340380 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.340356    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"]
Jul 21 18:18:48.342489 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:18:48.342459    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66de434f_8eae_4260_95b1_11ddc5cd922b.slice/crio-fd4cfde4810f8bee1bb7c789fa583106587badeacadc745ddfb70e379b4e1e3e WatchSource:0}: Error finding container fd4cfde4810f8bee1bb7c789fa583106587badeacadc745ddfb70e379b4e1e3e: Status 404 returned error can't find the container with id fd4cfde4810f8bee1bb7c789fa583106587badeacadc745ddfb70e379b4e1e3e
Jul 21 18:18:48.573069 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.573030    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" event={"ID":"66de434f-8eae-4260-95b1-11ddc5cd922b","Type":"ContainerStarted","Data":"bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f"}
Jul 21 18:18:48.573460 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.573093    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" event={"ID":"66de434f-8eae-4260-95b1-11ddc5cd922b","Type":"ContainerStarted","Data":"fd4cfde4810f8bee1bb7c789fa583106587badeacadc745ddfb70e379b4e1e3e"}
Jul 21 18:18:48.910189 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:48.910113    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="15e3e147-43ec-48d8-b8b2-61d307983a80" path="/var/lib/kubelet/pods/15e3e147-43ec-48d8-b8b2-61d307983a80/volumes"
Jul 21 18:18:52.590755 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:52.590662    2563 generic.go:356] "Generic (PLEG): container finished" podID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerID="bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f" exitCode=0
Jul 21 18:18:52.590755 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:52.590738    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" event={"ID":"66de434f-8eae-4260-95b1-11ddc5cd922b","Type":"ContainerDied","Data":"bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f"}
Jul 21 18:18:53.598082 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:53.598047    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" event={"ID":"66de434f-8eae-4260-95b1-11ddc5cd922b","Type":"ContainerStarted","Data":"af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40"}
Jul 21 18:18:53.598525 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:53.598109    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" event={"ID":"66de434f-8eae-4260-95b1-11ddc5cd922b","Type":"ContainerStarted","Data":"ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5"}
Jul 21 18:18:53.598525 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:53.598427    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:53.598620 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:53.598556    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:53.600033 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:53.600007    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:18:53.618575 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:53.618523    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podStartSLOduration=2.791043388 podStartE2EDuration="6.618508349s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.827464961s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:18:53.616263593 +0000 UTC m=+3193.174270699" watchObservedRunningTime="2026-07-21 18:18:53.618508349 +0000 UTC m=+3193.176515455"
Jul 21 18:18:54.602983 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:54.602942    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:18:59.607024 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:59.606987    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:18:59.607619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:18:59.607545    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:19:09.608010 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:19:09.607970    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:19:19.608096 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:19:19.608053    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:19:29.607560 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:19:29.607521    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:19:39.607595 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:19:39.607512    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:19:49.608119 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:19:49.608077    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:19:59.607702 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:19:59.607660    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:20:09.607961 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:09.607933    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:20:18.081258 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:18.081225    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"]
Jul 21 18:20:18.081763 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:18.081474    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" containerID="cri-o://ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5" gracePeriod=30
Jul 21 18:20:18.081763 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:18.081645    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kube-rbac-proxy" containerID="cri-o://af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40" gracePeriod=30
Jul 21 18:20:18.973635 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:18.973598    2563 generic.go:356] "Generic (PLEG): container finished" podID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerID="af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40" exitCode=2
Jul 21 18:20:18.973839 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:18.973666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" event={"ID":"66de434f-8eae-4260-95b1-11ddc5cd922b","Type":"ContainerDied","Data":"af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40"}
Jul 21 18:20:19.603502 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:19.603461    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 21 18:20:19.607739 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:19.607713    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 21 18:20:23.236895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.236863    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"]
Jul 21 18:20:23.240021 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.240004    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.242492 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.242469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 21 18:20:23.242619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.242474    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 21 18:20:23.250487 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.250461    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"]
Jul 21 18:20:23.354156 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.354092    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a471b50-0be9-4380-a669-be98f16f6db1-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.354156 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.354130    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.354156 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.354147    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5gt4k\" (UniqueName: \"kubernetes.io/projected/7a471b50-0be9-4380-a669-be98f16f6db1-kube-api-access-5gt4k\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.354338 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.354247    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.354338 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.354303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a471b50-0be9-4380-a669-be98f16f6db1-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.455313 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.455285    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.455477 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.455330    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a471b50-0be9-4380-a669-be98f16f6db1-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.455477 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.455389    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a471b50-0be9-4380-a669-be98f16f6db1-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.455477 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.455416    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.455477 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.455454    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5gt4k\" (UniqueName: \"kubernetes.io/projected/7a471b50-0be9-4380-a669-be98f16f6db1-kube-api-access-5gt4k\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.455905 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.455882    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a471b50-0be9-4380-a669-be98f16f6db1-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.456076 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.456057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.456135 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.456082    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.458115 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.458093    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a471b50-0be9-4380-a669-be98f16f6db1-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.463261 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.463235    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5gt4k\" (UniqueName: \"kubernetes.io/projected/7a471b50-0be9-4380-a669-be98f16f6db1-kube-api-access-5gt4k\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.550317 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.550284    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:23.737574 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.737551    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:20:23.859446 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.859412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-rbac-proxy-sar-config\") pod \"66de434f-8eae-4260-95b1-11ddc5cd922b\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") "
Jul 21 18:20:23.859643 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.859469    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66de434f-8eae-4260-95b1-11ddc5cd922b-kserve-provision-location\") pod \"66de434f-8eae-4260-95b1-11ddc5cd922b\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") "
Jul 21 18:20:23.859643 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.859494    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-cabundle-cert\") pod \"66de434f-8eae-4260-95b1-11ddc5cd922b\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") "
Jul 21 18:20:23.859643 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.859521    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66de434f-8eae-4260-95b1-11ddc5cd922b-proxy-tls\") pod \"66de434f-8eae-4260-95b1-11ddc5cd922b\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") "
Jul 21 18:20:23.859643 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.859543    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q8pv7\" (UniqueName: \"kubernetes.io/projected/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-api-access-q8pv7\") pod \"66de434f-8eae-4260-95b1-11ddc5cd922b\" (UID: \"66de434f-8eae-4260-95b1-11ddc5cd922b\") "
Jul 21 18:20:23.859904 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.859849    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/66de434f-8eae-4260-95b1-11ddc5cd922b-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "66de434f-8eae-4260-95b1-11ddc5cd922b" (UID: "66de434f-8eae-4260-95b1-11ddc5cd922b"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:20:23.859904 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.859866    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "66de434f-8eae-4260-95b1-11ddc5cd922b" (UID: "66de434f-8eae-4260-95b1-11ddc5cd922b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:20:23.859904 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.859900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "66de434f-8eae-4260-95b1-11ddc5cd922b" (UID: "66de434f-8eae-4260-95b1-11ddc5cd922b"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:20:23.861774 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.861742    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66de434f-8eae-4260-95b1-11ddc5cd922b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "66de434f-8eae-4260-95b1-11ddc5cd922b" (UID: "66de434f-8eae-4260-95b1-11ddc5cd922b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:20:23.861904 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.861809    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-api-access-q8pv7" (OuterVolumeSpecName: "kube-api-access-q8pv7") pod "66de434f-8eae-4260-95b1-11ddc5cd922b" (UID: "66de434f-8eae-4260-95b1-11ddc5cd922b"). InnerVolumeSpecName "kube-api-access-q8pv7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:20:23.895428 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.895380    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"]
Jul 21 18:20:23.897748 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:20:23.897722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7a471b50_0be9_4380_a669_be98f16f6db1.slice/crio-e4c3880b2081a0b1f6a66f1c756db28a6012238713d5cca19c13bcd9a753b2c2 WatchSource:0}: Error finding container e4c3880b2081a0b1f6a66f1c756db28a6012238713d5cca19c13bcd9a753b2c2: Status 404 returned error can't find the container with id e4c3880b2081a0b1f6a66f1c756db28a6012238713d5cca19c13bcd9a753b2c2
Jul 21 18:20:23.960758 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.960732    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:20:23.960758 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.960759    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/66de434f-8eae-4260-95b1-11ddc5cd922b-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:20:23.960988 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.960772    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/66de434f-8eae-4260-95b1-11ddc5cd922b-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:20:23.960988 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.960807    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/66de434f-8eae-4260-95b1-11ddc5cd922b-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:20:23.960988 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.960819    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8pv7\" (UniqueName: \"kubernetes.io/projected/66de434f-8eae-4260-95b1-11ddc5cd922b-kube-api-access-q8pv7\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:20:23.997275 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.997246    2563 generic.go:356] "Generic (PLEG): container finished" podID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerID="ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5" exitCode=0
Jul 21 18:20:23.997424 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.997308    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"
Jul 21 18:20:23.997424 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.997336    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" event={"ID":"66de434f-8eae-4260-95b1-11ddc5cd922b","Type":"ContainerDied","Data":"ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5"}
Jul 21 18:20:23.997424 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.997398    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4" event={"ID":"66de434f-8eae-4260-95b1-11ddc5cd922b","Type":"ContainerDied","Data":"fd4cfde4810f8bee1bb7c789fa583106587badeacadc745ddfb70e379b4e1e3e"}
Jul 21 18:20:23.997562 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.997439    2563 scope.go:117] "RemoveContainer" containerID="af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40"
Jul 21 18:20:23.999139 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.999117    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" event={"ID":"7a471b50-0be9-4380-a669-be98f16f6db1","Type":"ContainerStarted","Data":"8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf"}
Jul 21 18:20:23.999249 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:23.999172    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" event={"ID":"7a471b50-0be9-4380-a669-be98f16f6db1","Type":"ContainerStarted","Data":"e4c3880b2081a0b1f6a66f1c756db28a6012238713d5cca19c13bcd9a753b2c2"}
Jul 21 18:20:24.007572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.007548    2563 scope.go:117] "RemoveContainer" containerID="ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5"
Jul 21 18:20:24.016688 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.016667    2563 scope.go:117] "RemoveContainer" containerID="bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f"
Jul 21 18:20:24.024783 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.024764    2563 scope.go:117] "RemoveContainer" containerID="af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40"
Jul 21 18:20:24.025131 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:20:24.025111    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40\": container with ID starting with af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40 not found: ID does not exist" containerID="af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40"
Jul 21 18:20:24.025206 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.025138    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40"} err="failed to get container status \"af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40\": rpc error: code = NotFound desc = could not find container \"af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40\": container with ID starting with af02570a5e2c3243a1c4783a61b8b8c6515ae204abe9817a2b3f0aab0b43cd40 not found: ID does not exist"
Jul 21 18:20:24.025206 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.025155    2563 scope.go:117] "RemoveContainer" containerID="ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5"
Jul 21 18:20:24.025418 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:20:24.025398    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5\": container with ID starting with ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5 not found: ID does not exist" containerID="ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5"
Jul 21 18:20:24.025462 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.025425    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5"} err="failed to get container status \"ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5\": rpc error: code = NotFound desc = could not find container \"ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5\": container with ID starting with ff3a6f4908e1aa62676e5cb5b87bd498a6d57d92b1ba771d6e55d77bb20a85b5 not found: ID does not exist"
Jul 21 18:20:24.025462 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.025442    2563 scope.go:117] "RemoveContainer" containerID="bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f"
Jul 21 18:20:24.025713 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:20:24.025688    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f\": container with ID starting with bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f not found: ID does not exist" containerID="bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f"
Jul 21 18:20:24.025768 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.025722    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f"} err="failed to get container status \"bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f\": rpc error: code = NotFound desc = could not find container \"bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f\": container with ID starting with bc6ad4a26ffbe840e6c147707fae51a8d94223765a789900718147bc7431945f not found: ID does not exist"
Jul 21 18:20:24.032263 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.032227    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"]
Jul 21 18:20:24.034523 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.034500    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-msdx4"]
Jul 21 18:20:24.908560 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:24.908528    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" path="/var/lib/kubelet/pods/66de434f-8eae-4260-95b1-11ddc5cd922b/volumes"
Jul 21 18:20:28.018680 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:28.018648    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a471b50-0be9-4380-a669-be98f16f6db1" containerID="8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf" exitCode=0
Jul 21 18:20:28.019079 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:28.018708    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" event={"ID":"7a471b50-0be9-4380-a669-be98f16f6db1","Type":"ContainerDied","Data":"8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf"}
Jul 21 18:20:29.026367 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:29.026331    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" event={"ID":"7a471b50-0be9-4380-a669-be98f16f6db1","Type":"ContainerStarted","Data":"dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52"}
Jul 21 18:20:29.026776 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:29.026395    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" event={"ID":"7a471b50-0be9-4380-a669-be98f16f6db1","Type":"ContainerStarted","Data":"57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2"}
Jul 21 18:20:29.026776 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:29.026601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:29.047451 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:29.047393    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podStartSLOduration=2.477957986 podStartE2EDuration="6.047374674s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.569416688s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:20:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:20:29.045885963 +0000 UTC m=+3288.603893091" watchObservedRunningTime="2026-07-21 18:20:29.047374674 +0000 UTC m=+3288.605381781"
Jul 21 18:20:30.031832 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:30.031783    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:36.043058 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:36.043030    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:20:42.967400 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:42.967368    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:20:42.967903 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:20:42.967883    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:21:06.044826 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:21:06.044726    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 18:21:16.043808 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:21:16.043756    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 18:21:26.044357 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:21:26.044321    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 18:21:36.043709 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:21:36.043664    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 18:21:47.085948 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:21:47.085906    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 21 18:21:56.045581 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:21:56.045546    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:22:03.354089 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:03.354053    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"]
Jul 21 18:22:03.354561 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:03.354342    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container" containerID="cri-o://57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2" gracePeriod=30
Jul 21 18:22:03.354642 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:03.354535    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kube-rbac-proxy" containerID="cri-o://dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52" gracePeriod=30
Jul 21 18:22:04.435835 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:04.435773    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a471b50-0be9-4380-a669-be98f16f6db1" containerID="dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52" exitCode=2
Jul 21 18:22:04.436221 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:04.435842    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" event={"ID":"7a471b50-0be9-4380-a669-be98f16f6db1","Type":"ContainerDied","Data":"dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52"}
Jul 21 18:22:06.037754 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:06.037709    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 21 18:22:06.044263 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:06.044228    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.37:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 21 18:22:08.095297 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.095267    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:22:08.126390 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.126361    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a471b50-0be9-4380-a669-be98f16f6db1-kserve-provision-location\") pod \"7a471b50-0be9-4380-a669-be98f16f6db1\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") "
Jul 21 18:22:08.126390 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.126399    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-kube-rbac-proxy-sar-config\") pod \"7a471b50-0be9-4380-a669-be98f16f6db1\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") "
Jul 21 18:22:08.126656 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.126446    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a471b50-0be9-4380-a669-be98f16f6db1-proxy-tls\") pod \"7a471b50-0be9-4380-a669-be98f16f6db1\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") "
Jul 21 18:22:08.126656 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.126481    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5gt4k\" (UniqueName: \"kubernetes.io/projected/7a471b50-0be9-4380-a669-be98f16f6db1-kube-api-access-5gt4k\") pod \"7a471b50-0be9-4380-a669-be98f16f6db1\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") "
Jul 21 18:22:08.126656 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.126516    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-cabundle-cert\") pod \"7a471b50-0be9-4380-a669-be98f16f6db1\" (UID: \"7a471b50-0be9-4380-a669-be98f16f6db1\") "
Jul 21 18:22:08.126873 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.126753    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7a471b50-0be9-4380-a669-be98f16f6db1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7a471b50-0be9-4380-a669-be98f16f6db1" (UID: "7a471b50-0be9-4380-a669-be98f16f6db1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:22:08.126946 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.126918    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7a471b50-0be9-4380-a669-be98f16f6db1" (UID: "7a471b50-0be9-4380-a669-be98f16f6db1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:22:08.127009 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.126943    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7a471b50-0be9-4380-a669-be98f16f6db1" (UID: "7a471b50-0be9-4380-a669-be98f16f6db1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:22:08.128835 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.128783    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7a471b50-0be9-4380-a669-be98f16f6db1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7a471b50-0be9-4380-a669-be98f16f6db1" (UID: "7a471b50-0be9-4380-a669-be98f16f6db1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:22:08.128943 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.128854    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7a471b50-0be9-4380-a669-be98f16f6db1-kube-api-access-5gt4k" (OuterVolumeSpecName: "kube-api-access-5gt4k") pod "7a471b50-0be9-4380-a669-be98f16f6db1" (UID: "7a471b50-0be9-4380-a669-be98f16f6db1"). InnerVolumeSpecName "kube-api-access-5gt4k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:22:08.227455 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.227367    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7a471b50-0be9-4380-a669-be98f16f6db1-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:22:08.227455 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.227399    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:22:08.227455 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.227411    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7a471b50-0be9-4380-a669-be98f16f6db1-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:22:08.227455 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.227421    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5gt4k\" (UniqueName: \"kubernetes.io/projected/7a471b50-0be9-4380-a669-be98f16f6db1-kube-api-access-5gt4k\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:22:08.227455 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.227430    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7a471b50-0be9-4380-a669-be98f16f6db1-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:22:08.454726 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.454694    2563 generic.go:356] "Generic (PLEG): container finished" podID="7a471b50-0be9-4380-a669-be98f16f6db1" containerID="57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2" exitCode=0
Jul 21 18:22:08.454928 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.454759    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"
Jul 21 18:22:08.454928 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.454768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" event={"ID":"7a471b50-0be9-4380-a669-be98f16f6db1","Type":"ContainerDied","Data":"57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2"}
Jul 21 18:22:08.454928 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.454857    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7" event={"ID":"7a471b50-0be9-4380-a669-be98f16f6db1","Type":"ContainerDied","Data":"e4c3880b2081a0b1f6a66f1c756db28a6012238713d5cca19c13bcd9a753b2c2"}
Jul 21 18:22:08.454928 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.454886    2563 scope.go:117] "RemoveContainer" containerID="dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52"
Jul 21 18:22:08.464185 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.464160    2563 scope.go:117] "RemoveContainer" containerID="57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2"
Jul 21 18:22:08.473022 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.472976    2563 scope.go:117] "RemoveContainer" containerID="8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf"
Jul 21 18:22:08.482005 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.481979    2563 scope.go:117] "RemoveContainer" containerID="dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52"
Jul 21 18:22:08.482458 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:22:08.482428    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52\": container with ID starting with dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52 not found: ID does not exist" containerID="dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52"
Jul 21 18:22:08.482568 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.482470    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52"} err="failed to get container status \"dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52\": rpc error: code = NotFound desc = could not find container \"dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52\": container with ID starting with dbdf5f87abd8a9528d42e9356adddff826327ce5c30e4d3c80ece8728d0faa52 not found: ID does not exist"
Jul 21 18:22:08.482568 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.482499    2563 scope.go:117] "RemoveContainer" containerID="57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2"
Jul 21 18:22:08.482866 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:22:08.482840    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2\": container with ID starting with 57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2 not found: ID does not exist" containerID="57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2"
Jul 21 18:22:08.482942 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.482874    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2"} err="failed to get container status \"57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2\": rpc error: code = NotFound desc = could not find container \"57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2\": container with ID starting with 57042737ed306861ffcbcaa7549a7082493605f2e40dcafdf8d51561c7bc91f2 not found: ID does not exist"
Jul 21 18:22:08.482942 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.482896    2563 scope.go:117] "RemoveContainer" containerID="8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf"
Jul 21 18:22:08.483195 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:22:08.483174    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf\": container with ID starting with 8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf not found: ID does not exist" containerID="8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf"
Jul 21 18:22:08.483252 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.483204    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf"} err="failed to get container status \"8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf\": rpc error: code = NotFound desc = could not find container \"8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf\": container with ID starting with 8b441070bee5348f8ea954aa2de262b3f33d678204856dd1a4c462ed3454bcdf not found: ID does not exist"
Jul 21 18:22:08.484230 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.484207    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"]
Jul 21 18:22:08.487695 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.487667    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-lcds7"]
Jul 21 18:22:08.539479 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539445    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"]
Jul 21 18:22:08.539841 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539824    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="storage-initializer"
Jul 21 18:22:08.539895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539845    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="storage-initializer"
Jul 21 18:22:08.539895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539865    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kube-rbac-proxy"
Jul 21 18:22:08.539895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539870    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kube-rbac-proxy"
Jul 21 18:22:08.539895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539877    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kube-rbac-proxy"
Jul 21 18:22:08.539895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539883    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kube-rbac-proxy"
Jul 21 18:22:08.539895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539892    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container"
Jul 21 18:22:08.539895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539897    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container"
Jul 21 18:22:08.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539904    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container"
Jul 21 18:22:08.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539910    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container"
Jul 21 18:22:08.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539931    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="storage-initializer"
Jul 21 18:22:08.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539936    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="storage-initializer"
Jul 21 18:22:08.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.539986    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kube-rbac-proxy"
Jul 21 18:22:08.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.540000    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="66de434f-8eae-4260-95b1-11ddc5cd922b" containerName="kserve-container"
Jul 21 18:22:08.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.540012    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kserve-container"
Jul 21 18:22:08.540123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.540018    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" containerName="kube-rbac-proxy"
Jul 21 18:22:08.544837 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.544817    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.547619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.547593    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 21 18:22:08.547751 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.547646    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:22:08.547751 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.547665    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:22:08.547751 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.547681    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:22:08.547751 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.547601    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 21 18:22:08.547751 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.547685    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:22:08.548076 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.548062    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:22:08.553748 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.553725    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"]
Jul 21 18:22:08.630640 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.630603    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hrkkg\" (UniqueName: \"kubernetes.io/projected/62a64646-ad24-4df7-babd-422f7c295a6a-kube-api-access-hrkkg\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.630876 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.630647    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.630876 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.630685    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a64646-ad24-4df7-babd-422f7c295a6a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.630876 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.630777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.630876 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.630848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.731456 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.731399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hrkkg\" (UniqueName: \"kubernetes.io/projected/62a64646-ad24-4df7-babd-422f7c295a6a-kube-api-access-hrkkg\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.731456 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.731453    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.731704 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.731480    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a64646-ad24-4df7-babd-422f7c295a6a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.731704 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.731537    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.731704 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.731555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.731704 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:22:08.731656    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-serving-cert: secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 21 18:22:08.731966 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:22:08.731732    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls podName:62a64646-ad24-4df7-babd-422f7c295a6a nodeName:}" failed. No retries permitted until 2026-07-21 18:22:09.231709561 +0000 UTC m=+3388.789716663 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls") pod "isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" (UID: "62a64646-ad24-4df7-babd-422f7c295a6a") : secret "isvc-predictive-xgboost-v2-predictor-serving-cert" not found
Jul 21 18:22:08.732022 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.731973    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a64646-ad24-4df7-babd-422f7c295a6a-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.732193 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.732176    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.732242 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.732225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.743455 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.743418    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hrkkg\" (UniqueName: \"kubernetes.io/projected/62a64646-ad24-4df7-babd-422f7c295a6a-kube-api-access-hrkkg\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:08.908960 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:08.908925    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7a471b50-0be9-4380-a669-be98f16f6db1" path="/var/lib/kubelet/pods/7a471b50-0be9-4380-a669-be98f16f6db1/volumes"
Jul 21 18:22:09.236187 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:09.236141    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:09.238680 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:09.238654    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:09.455665 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:09.455624    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:09.602481 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:09.602451    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"]
Jul 21 18:22:09.604981 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:22:09.604954    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62a64646_ad24_4df7_babd_422f7c295a6a.slice/crio-e2b8aee3462e35b57abb00e5bf5b99a2dfaaad9d82456c1db34fa4cdb528215f WatchSource:0}: Error finding container e2b8aee3462e35b57abb00e5bf5b99a2dfaaad9d82456c1db34fa4cdb528215f: Status 404 returned error can't find the container with id e2b8aee3462e35b57abb00e5bf5b99a2dfaaad9d82456c1db34fa4cdb528215f
Jul 21 18:22:10.467700 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:10.467661    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" event={"ID":"62a64646-ad24-4df7-babd-422f7c295a6a","Type":"ContainerStarted","Data":"7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b"}
Jul 21 18:22:10.468155 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:10.467745    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" event={"ID":"62a64646-ad24-4df7-babd-422f7c295a6a","Type":"ContainerStarted","Data":"e2b8aee3462e35b57abb00e5bf5b99a2dfaaad9d82456c1db34fa4cdb528215f"}
Jul 21 18:22:13.484595 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:13.484559    2563 generic.go:356] "Generic (PLEG): container finished" podID="62a64646-ad24-4df7-babd-422f7c295a6a" containerID="7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b" exitCode=0
Jul 21 18:22:13.485013 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:13.484638    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" event={"ID":"62a64646-ad24-4df7-babd-422f7c295a6a","Type":"ContainerDied","Data":"7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b"}
Jul 21 18:22:14.491970 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:14.491933    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" event={"ID":"62a64646-ad24-4df7-babd-422f7c295a6a","Type":"ContainerStarted","Data":"e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891"}
Jul 21 18:22:14.492376 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:14.492000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" event={"ID":"62a64646-ad24-4df7-babd-422f7c295a6a","Type":"ContainerStarted","Data":"d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5"}
Jul 21 18:22:14.492376 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:14.492325    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:14.492376 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:14.492374    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:14.514510 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:14.514459    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podStartSLOduration=2.865751588 podStartE2EDuration="6.514441389s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.648689801s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:22:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:22:14.513164915 +0000 UTC m=+3394.071172030" watchObservedRunningTime="2026-07-21 18:22:14.514441389 +0000 UTC m=+3394.072448568"
Jul 21 18:22:20.502475 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:20.502445    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:22:50.504176 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:22:50.504129    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 18:23:00.503424 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:00.503378    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 18:23:10.503887 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:10.503842    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 18:23:20.503723 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:20.503677    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 18:23:30.530015 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:30.529978    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:23:38.651688 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:38.651647    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"]
Jul 21 18:23:38.652162 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:38.652024    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container" containerID="cri-o://d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5" gracePeriod=30
Jul 21 18:23:38.652162 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:38.652054    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kube-rbac-proxy" containerID="cri-o://e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891" gracePeriod=30
Jul 21 18:23:38.857203 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:38.857172    2563 generic.go:356] "Generic (PLEG): container finished" podID="62a64646-ad24-4df7-babd-422f7c295a6a" containerID="e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891" exitCode=2
Jul 21 18:23:38.857385 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:38.857253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" event={"ID":"62a64646-ad24-4df7-babd-422f7c295a6a","Type":"ContainerDied","Data":"e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891"}
Jul 21 18:23:40.498165 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:40.498120    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 21 18:23:40.503653 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:40.503616    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.38:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 21 18:23:43.607511 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.607484    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:23:43.655036 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.654989    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls\") pod \"62a64646-ad24-4df7-babd-422f7c295a6a\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") "
Jul 21 18:23:43.655239 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.655081    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a64646-ad24-4df7-babd-422f7c295a6a-kserve-provision-location\") pod \"62a64646-ad24-4df7-babd-422f7c295a6a\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") "
Jul 21 18:23:43.655239 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.655171    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hrkkg\" (UniqueName: \"kubernetes.io/projected/62a64646-ad24-4df7-babd-422f7c295a6a-kube-api-access-hrkkg\") pod \"62a64646-ad24-4df7-babd-422f7c295a6a\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") "
Jul 21 18:23:43.655239 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.655201    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-cabundle-cert\") pod \"62a64646-ad24-4df7-babd-422f7c295a6a\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") "
Jul 21 18:23:43.655426 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.655258    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-kube-rbac-proxy-sar-config\") pod \"62a64646-ad24-4df7-babd-422f7c295a6a\" (UID: \"62a64646-ad24-4df7-babd-422f7c295a6a\") "
Jul 21 18:23:43.655426 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.655361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62a64646-ad24-4df7-babd-422f7c295a6a-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "62a64646-ad24-4df7-babd-422f7c295a6a" (UID: "62a64646-ad24-4df7-babd-422f7c295a6a"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:23:43.655584 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.655566    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/62a64646-ad24-4df7-babd-422f7c295a6a-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:23:43.655645 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.655608    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "62a64646-ad24-4df7-babd-422f7c295a6a" (UID: "62a64646-ad24-4df7-babd-422f7c295a6a"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:23:43.655645 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.655631    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "62a64646-ad24-4df7-babd-422f7c295a6a" (UID: "62a64646-ad24-4df7-babd-422f7c295a6a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:23:43.657448 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.657423    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62a64646-ad24-4df7-babd-422f7c295a6a-kube-api-access-hrkkg" (OuterVolumeSpecName: "kube-api-access-hrkkg") pod "62a64646-ad24-4df7-babd-422f7c295a6a" (UID: "62a64646-ad24-4df7-babd-422f7c295a6a"). InnerVolumeSpecName "kube-api-access-hrkkg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:23:43.657673 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.657648    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "62a64646-ad24-4df7-babd-422f7c295a6a" (UID: "62a64646-ad24-4df7-babd-422f7c295a6a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:23:43.757062 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.757020    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/62a64646-ad24-4df7-babd-422f7c295a6a-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:23:43.757062 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.757058    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hrkkg\" (UniqueName: \"kubernetes.io/projected/62a64646-ad24-4df7-babd-422f7c295a6a-kube-api-access-hrkkg\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:23:43.757062 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.757069    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:23:43.757320 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.757078    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/62a64646-ad24-4df7-babd-422f7c295a6a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:23:43.803341 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803305    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"]
Jul 21 18:23:43.803698 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803683    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kube-rbac-proxy"
Jul 21 18:23:43.803781 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803701    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kube-rbac-proxy"
Jul 21 18:23:43.803781 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803718    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="storage-initializer"
Jul 21 18:23:43.803781 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803725    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="storage-initializer"
Jul 21 18:23:43.803781 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803732    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container"
Jul 21 18:23:43.803781 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803737    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container"
Jul 21 18:23:43.804012 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803827    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kube-rbac-proxy"
Jul 21 18:23:43.804012 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.803837    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" containerName="kserve-container"
Jul 21 18:23:43.807319 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.807299    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.809956 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.809937    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 21 18:23:43.809956 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.809949    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 21 18:23:43.819151 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.819122    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"]
Jul 21 18:23:43.858309 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.858259    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.858309 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.858312    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.858532 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.858395    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.858532 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.858458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whp9c\" (UniqueName: \"kubernetes.io/projected/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-api-access-whp9c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.858532 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.858482    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.884494 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.884460    2563 generic.go:356] "Generic (PLEG): container finished" podID="62a64646-ad24-4df7-babd-422f7c295a6a" containerID="d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5" exitCode=0
Jul 21 18:23:43.884494 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.884504    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" event={"ID":"62a64646-ad24-4df7-babd-422f7c295a6a","Type":"ContainerDied","Data":"d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5"}
Jul 21 18:23:43.884774 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.884522    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"
Jul 21 18:23:43.884774 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.884570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk" event={"ID":"62a64646-ad24-4df7-babd-422f7c295a6a","Type":"ContainerDied","Data":"e2b8aee3462e35b57abb00e5bf5b99a2dfaaad9d82456c1db34fa4cdb528215f"}
Jul 21 18:23:43.884774 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.884606    2563 scope.go:117] "RemoveContainer" containerID="e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891"
Jul 21 18:23:43.893722 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.893702    2563 scope.go:117] "RemoveContainer" containerID="d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5"
Jul 21 18:23:43.902046 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.902023    2563 scope.go:117] "RemoveContainer" containerID="7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b"
Jul 21 18:23:43.907064 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.907040    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"]
Jul 21 18:23:43.910822 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.910778    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-kltwk"]
Jul 21 18:23:43.911127 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.911113    2563 scope.go:117] "RemoveContainer" containerID="e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891"
Jul 21 18:23:43.911449 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:23:43.911426    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891\": container with ID starting with e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891 not found: ID does not exist" containerID="e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891"
Jul 21 18:23:43.911519 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.911458    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891"} err="failed to get container status \"e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891\": rpc error: code = NotFound desc = could not find container \"e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891\": container with ID starting with e28e0e836cd168bdc630ae8ccd612883a24a97e2d8a1e7cb47a1e55ddf9eb891 not found: ID does not exist"
Jul 21 18:23:43.911519 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.911480    2563 scope.go:117] "RemoveContainer" containerID="d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5"
Jul 21 18:23:43.911736 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:23:43.911718    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5\": container with ID starting with d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5 not found: ID does not exist" containerID="d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5"
Jul 21 18:23:43.911778 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.911742    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5"} err="failed to get container status \"d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5\": rpc error: code = NotFound desc = could not find container \"d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5\": container with ID starting with d0dcfd6a31867ec07c039141cf9522bc4d7e00e4d45f2da098f6891df8be87e5 not found: ID does not exist"
Jul 21 18:23:43.911778 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.911756    2563 scope.go:117] "RemoveContainer" containerID="7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b"
Jul 21 18:23:43.912018 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:23:43.912000    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b\": container with ID starting with 7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b not found: ID does not exist" containerID="7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b"
Jul 21 18:23:43.912067 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.912023    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b"} err="failed to get container status \"7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b\": rpc error: code = NotFound desc = could not find container \"7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b\": container with ID starting with 7cfb3508445e7f0f8ef6110526bbf1c8e0367579e13af96e90b5de292e31cc5b not found: ID does not exist"
Jul 21 18:23:43.959628 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.959586    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whp9c\" (UniqueName: \"kubernetes.io/projected/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-api-access-whp9c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.959851 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.959632    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.959851 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.959683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.959851 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.959719    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.959851 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.959749    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.960086 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:23:43.959873    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-serving-cert: secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 21 18:23:43.960086 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:23:43.959951    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls podName:cde0088e-32f8-4dd2-9a01-0f8acbf32705 nodeName:}" failed. No retries permitted until 2026-07-21 18:23:44.45992908 +0000 UTC m=+3484.017936168 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls") pod "isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" (UID: "cde0088e-32f8-4dd2-9a01-0f8acbf32705") : secret "isvc-predictive-lightgbm-v2-predictor-serving-cert" not found
Jul 21 18:23:43.960259 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.960239    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.960491 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.960471    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.960491 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.960485    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:43.970450 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:43.970419    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whp9c\" (UniqueName: \"kubernetes.io/projected/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-api-access-whp9c\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:44.464523 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:44.464477    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:44.467184 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:44.467154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:44.718027 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:44.717922    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:44.859129 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:44.859100    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"]
Jul 21 18:23:44.861700 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:23:44.861671    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcde0088e_32f8_4dd2_9a01_0f8acbf32705.slice/crio-d9f7849e1c8ba97d79e3039687063aee0ca47818f11d40b02c309a65a8b04bb0 WatchSource:0}: Error finding container d9f7849e1c8ba97d79e3039687063aee0ca47818f11d40b02c309a65a8b04bb0: Status 404 returned error can't find the container with id d9f7849e1c8ba97d79e3039687063aee0ca47818f11d40b02c309a65a8b04bb0
Jul 21 18:23:44.891468 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:44.891435    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" event={"ID":"cde0088e-32f8-4dd2-9a01-0f8acbf32705","Type":"ContainerStarted","Data":"d9f7849e1c8ba97d79e3039687063aee0ca47818f11d40b02c309a65a8b04bb0"}
Jul 21 18:23:44.908589 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:44.908566    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62a64646-ad24-4df7-babd-422f7c295a6a" path="/var/lib/kubelet/pods/62a64646-ad24-4df7-babd-422f7c295a6a/volumes"
Jul 21 18:23:45.900435 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:45.900393    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" event={"ID":"cde0088e-32f8-4dd2-9a01-0f8acbf32705","Type":"ContainerStarted","Data":"c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a"}
Jul 21 18:23:48.916878 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:48.916775    2563 generic.go:356] "Generic (PLEG): container finished" podID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerID="c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a" exitCode=0
Jul 21 18:23:48.916878 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:48.916849    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" event={"ID":"cde0088e-32f8-4dd2-9a01-0f8acbf32705","Type":"ContainerDied","Data":"c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a"}
Jul 21 18:23:49.923656 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:49.923617    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" event={"ID":"cde0088e-32f8-4dd2-9a01-0f8acbf32705","Type":"ContainerStarted","Data":"f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3"}
Jul 21 18:23:49.924117 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:49.923703    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" event={"ID":"cde0088e-32f8-4dd2-9a01-0f8acbf32705","Type":"ContainerStarted","Data":"470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071"}
Jul 21 18:23:49.924117 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:49.923940    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:49.946609 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:49.946546    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podStartSLOduration=3.209205097 podStartE2EDuration="6.946527951s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.737322854s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:23:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:23:49.945235148 +0000 UTC m=+3489.503242256" watchObservedRunningTime="2026-07-21 18:23:49.946527951 +0000 UTC m=+3489.504535058"
Jul 21 18:23:50.928629 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:50.928598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:23:56.940338 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:23:56.940303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:24:26.942090 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:24:26.941996    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 18:24:36.942024 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:24:36.941971    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 18:24:46.941743 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:24:46.941691    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 18:24:56.941664 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:24:56.941623    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 18:25:06.931844 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:06.931805    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:25:13.916236 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:13.916200    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"]
Jul 21 18:25:13.916766 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:13.916428    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container" containerID="cri-o://470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071" gracePeriod=30
Jul 21 18:25:13.916766 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:13.916495    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kube-rbac-proxy" containerID="cri-o://f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3" gracePeriod=30
Jul 21 18:25:14.312313 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:14.312277    2563 generic.go:356] "Generic (PLEG): container finished" podID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerID="f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3" exitCode=2
Jul 21 18:25:14.312523 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:14.312348    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" event={"ID":"cde0088e-32f8-4dd2-9a01-0f8acbf32705","Type":"ContainerDied","Data":"f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3"}
Jul 21 18:25:16.903848 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:16.903769    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.39:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 21 18:25:16.934522 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:16.934471    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 21 18:25:20.166180 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.166153    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:25:20.231363 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.231314    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls\") pod \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") "
Jul 21 18:25:20.231579 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.231392    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-rbac-proxy-sar-config\") pod \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") "
Jul 21 18:25:20.231579 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.231473    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whp9c\" (UniqueName: \"kubernetes.io/projected/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-api-access-whp9c\") pod \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") "
Jul 21 18:25:20.231579 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.231497    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kserve-provision-location\") pod \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") "
Jul 21 18:25:20.231579 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.231531    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-cabundle-cert\") pod \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\" (UID: \"cde0088e-32f8-4dd2-9a01-0f8acbf32705\") "
Jul 21 18:25:20.231831 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.231801    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "cde0088e-32f8-4dd2-9a01-0f8acbf32705" (UID: "cde0088e-32f8-4dd2-9a01-0f8acbf32705"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:25:20.231898 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.231872    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "cde0088e-32f8-4dd2-9a01-0f8acbf32705" (UID: "cde0088e-32f8-4dd2-9a01-0f8acbf32705"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:25:20.231941 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.231908    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "cde0088e-32f8-4dd2-9a01-0f8acbf32705" (UID: "cde0088e-32f8-4dd2-9a01-0f8acbf32705"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:25:20.233531 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.233510    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "cde0088e-32f8-4dd2-9a01-0f8acbf32705" (UID: "cde0088e-32f8-4dd2-9a01-0f8acbf32705"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:25:20.233619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.233600    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-api-access-whp9c" (OuterVolumeSpecName: "kube-api-access-whp9c") pod "cde0088e-32f8-4dd2-9a01-0f8acbf32705" (UID: "cde0088e-32f8-4dd2-9a01-0f8acbf32705"). InnerVolumeSpecName "kube-api-access-whp9c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:25:20.332636 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.332538    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:25:20.332636 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.332572    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whp9c\" (UniqueName: \"kubernetes.io/projected/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kube-api-access-whp9c\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:25:20.332636 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.332584    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/cde0088e-32f8-4dd2-9a01-0f8acbf32705-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:25:20.332636 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.332594    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/cde0088e-32f8-4dd2-9a01-0f8acbf32705-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:25:20.332636 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.332603    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/cde0088e-32f8-4dd2-9a01-0f8acbf32705-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:25:20.339971 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.339938    2563 generic.go:356] "Generic (PLEG): container finished" podID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerID="470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071" exitCode=0
Jul 21 18:25:20.340128 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.340005    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"
Jul 21 18:25:20.340128 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.340018    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" event={"ID":"cde0088e-32f8-4dd2-9a01-0f8acbf32705","Type":"ContainerDied","Data":"470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071"}
Jul 21 18:25:20.340128 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.340085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw" event={"ID":"cde0088e-32f8-4dd2-9a01-0f8acbf32705","Type":"ContainerDied","Data":"d9f7849e1c8ba97d79e3039687063aee0ca47818f11d40b02c309a65a8b04bb0"}
Jul 21 18:25:20.340128 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.340113    2563 scope.go:117] "RemoveContainer" containerID="f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3"
Jul 21 18:25:20.350223 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.350204    2563 scope.go:117] "RemoveContainer" containerID="470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071"
Jul 21 18:25:20.358824 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.358799    2563 scope.go:117] "RemoveContainer" containerID="c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a"
Jul 21 18:25:20.363847 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.363821    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"]
Jul 21 18:25:20.367523 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.367498    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-b2jvw"]
Jul 21 18:25:20.368403 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.368378    2563 scope.go:117] "RemoveContainer" containerID="f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3"
Jul 21 18:25:20.368690 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:25:20.368672    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3\": container with ID starting with f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3 not found: ID does not exist" containerID="f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3"
Jul 21 18:25:20.368765 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.368700    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3"} err="failed to get container status \"f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3\": rpc error: code = NotFound desc = could not find container \"f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3\": container with ID starting with f24f2b2482553953fc6a669d0a53118be5ce4c6e717df1f9cf0ee711846552d3 not found: ID does not exist"
Jul 21 18:25:20.368765 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.368719    2563 scope.go:117] "RemoveContainer" containerID="470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071"
Jul 21 18:25:20.368978 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:25:20.368959    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071\": container with ID starting with 470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071 not found: ID does not exist" containerID="470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071"
Jul 21 18:25:20.369048 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.368989    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071"} err="failed to get container status \"470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071\": rpc error: code = NotFound desc = could not find container \"470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071\": container with ID starting with 470cfa325ef798b634fbd82de76a1b85098f0c6dda50401c7ad9c898239ff071 not found: ID does not exist"
Jul 21 18:25:20.369048 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.369012    2563 scope.go:117] "RemoveContainer" containerID="c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a"
Jul 21 18:25:20.369310 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:25:20.369295    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a\": container with ID starting with c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a not found: ID does not exist" containerID="c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a"
Jul 21 18:25:20.369359 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.369314    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a"} err="failed to get container status \"c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a\": rpc error: code = NotFound desc = could not find container \"c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a\": container with ID starting with c8512ecacee70a14653f94e1071c41c93f4a265db7b799617767af23b88b235a not found: ID does not exist"
Jul 21 18:25:20.908024 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:20.907983    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" path="/var/lib/kubelet/pods/cde0088e-32f8-4dd2-9a01-0f8acbf32705/volumes"
Jul 21 18:25:42.994981 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:42.994949    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:25:42.997046 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:25:42.997021    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:30:43.023313 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:30:43.023283    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:30:43.026223 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:30:43.026203    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:33:04.955272 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955190    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"]
Jul 21 18:33:04.957834 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955645    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kube-rbac-proxy"
Jul 21 18:33:04.957834 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955662    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kube-rbac-proxy"
Jul 21 18:33:04.957834 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955689    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container"
Jul 21 18:33:04.957834 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955696    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container"
Jul 21 18:33:04.957834 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955714    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="storage-initializer"
Jul 21 18:33:04.957834 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955719    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="storage-initializer"
Jul 21 18:33:04.957834 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955771    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kube-rbac-proxy"
Jul 21 18:33:04.957834 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.955778    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="cde0088e-32f8-4dd2-9a01-0f8acbf32705" containerName="kserve-container"
Jul 21 18:33:04.958781 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.958763    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:04.961569 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.961543    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-predictor-serving-cert\""
Jul 21 18:33:04.961700 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.961573    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:33:04.961700 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.961550    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:33:04.961700 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.961580    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:33:04.961700 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.961688    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:33:04.962302 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.962029    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:33:04.963046 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.963029    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-kube-rbac-proxy-sar-config\""
Jul 21 18:33:04.968680 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:04.968658    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"]
Jul 21 18:33:05.062703 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.062674    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.062862 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.062714    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pttw2\" (UniqueName: \"kubernetes.io/projected/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-api-access-pttw2\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.062862 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.062818    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02bcdad-f065-48d2-8537-fb491e25b93f-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.062945 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.062863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.062945 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.062912    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.163268 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.163236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02bcdad-f065-48d2-8537-fb491e25b93f-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.163421 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.163284    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.163421 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.163304    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.163421 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.163340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.163421 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.163364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pttw2\" (UniqueName: \"kubernetes.io/projected/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-api-access-pttw2\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.163421 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:33:05.163403    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-tensorflow-predictor-serving-cert: secret "isvc-tensorflow-predictor-serving-cert" not found
Jul 21 18:33:05.163650 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:33:05.163474    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls podName:b02bcdad-f065-48d2-8537-fb491e25b93f nodeName:}" failed. No retries permitted until 2026-07-21 18:33:05.663452844 +0000 UTC m=+4045.221459932 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls") pod "isvc-tensorflow-predictor-547dfc5669-9hbtt" (UID: "b02bcdad-f065-48d2-8537-fb491e25b93f") : secret "isvc-tensorflow-predictor-serving-cert" not found
Jul 21 18:33:05.163727 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.163675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02bcdad-f065-48d2-8537-fb491e25b93f-kserve-provision-location\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.164025 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.164009    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-cabundle-cert\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.164025 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.164019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.174726 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.174702    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pttw2\" (UniqueName: \"kubernetes.io/projected/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-api-access-pttw2\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.669655 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.669622    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.672171 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.672153    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls\") pod \"isvc-tensorflow-predictor-547dfc5669-9hbtt\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") " pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:05.869666 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:05.869631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:06.002730 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:06.002704    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"]
Jul 21 18:33:06.004892 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:33:06.004862    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb02bcdad_f065_48d2_8537_fb491e25b93f.slice/crio-d705f5dbef059ed8971e7a7b2665f8cd7099cc960b2d80c8dc66a73fca73e3a4 WatchSource:0}: Error finding container d705f5dbef059ed8971e7a7b2665f8cd7099cc960b2d80c8dc66a73fca73e3a4: Status 404 returned error can't find the container with id d705f5dbef059ed8971e7a7b2665f8cd7099cc960b2d80c8dc66a73fca73e3a4
Jul 21 18:33:06.307657 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:06.307615    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" event={"ID":"b02bcdad-f065-48d2-8537-fb491e25b93f","Type":"ContainerStarted","Data":"bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd"}
Jul 21 18:33:06.307882 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:06.307672    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" event={"ID":"b02bcdad-f065-48d2-8537-fb491e25b93f","Type":"ContainerStarted","Data":"d705f5dbef059ed8971e7a7b2665f8cd7099cc960b2d80c8dc66a73fca73e3a4"}
Jul 21 18:33:11.331978 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:11.331945    2563 generic.go:356] "Generic (PLEG): container finished" podID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerID="bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd" exitCode=0
Jul 21 18:33:11.332524 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:11.332024    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" event={"ID":"b02bcdad-f065-48d2-8537-fb491e25b93f","Type":"ContainerDied","Data":"bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd"}
Jul 21 18:33:11.333405 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:11.333387    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 18:33:15.354855 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:15.354824    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" event={"ID":"b02bcdad-f065-48d2-8537-fb491e25b93f","Type":"ContainerStarted","Data":"7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586"}
Jul 21 18:33:16.361279 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:16.361242    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" event={"ID":"b02bcdad-f065-48d2-8537-fb491e25b93f","Type":"ContainerStarted","Data":"8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86"}
Jul 21 18:33:16.361684 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:16.361318    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:16.361684 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:16.361343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:16.362673 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:16.362649    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 18:33:16.381399 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:16.381357    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podStartSLOduration=4.121623276 podStartE2EDuration="12.381343184s" totalImagesPullingTime="3.917773187s" totalInitContainerRuntime="4.341946721s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:33:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:33:16.380021339 +0000 UTC m=+4055.938028446" watchObservedRunningTime="2026-07-21 18:33:16.381343184 +0000 UTC m=+4055.939350292"
Jul 21 18:33:17.366281 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:17.366243    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 18:33:22.372876 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:22.372846    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:22.373454 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:22.373396    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 21 18:33:32.374214 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:32.374186    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:45.586854 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:45.586817    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"]
Jul 21 18:33:45.587361 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:45.587070    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kserve-container" containerID="cri-o://7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586" gracePeriod=30
Jul 21 18:33:45.587361 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:45.587136    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy" containerID="cri-o://8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86" gracePeriod=30
Jul 21 18:33:46.489628 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:46.489595    2563 generic.go:356] "Generic (PLEG): container finished" podID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerID="8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86" exitCode=2
Jul 21 18:33:46.489836 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:46.489666    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" event={"ID":"b02bcdad-f065-48d2-8537-fb491e25b93f","Type":"ContainerDied","Data":"8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86"}
Jul 21 18:33:47.366863 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:47.366779    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 21 18:33:50.762474 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.762430    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"]
Jul 21 18:33:50.765341 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.765319    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.767901 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.767877    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-kube-rbac-proxy-sar-config\""
Jul 21 18:33:50.768026 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.767926    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-tensorflow-runtime-predictor-serving-cert\""
Jul 21 18:33:50.778136 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.778108    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"]
Jul 21 18:33:50.856386 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.856346    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.856386 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.856383    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.856610 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.856413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.856610 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.856443    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.856610 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.856463    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpnst\" (UniqueName: \"kubernetes.io/projected/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-api-access-lpnst\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.957109 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.957076    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.957109 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.957108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.957363 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.957139    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.957363 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.957292    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.957363 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.957327    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpnst\" (UniqueName: \"kubernetes.io/projected/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-api-access-lpnst\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.957573 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.957549    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kserve-provision-location\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.957917 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.957896    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-cabundle-cert\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.958011 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.957993    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-rbac-proxy-sar-config\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.959840 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.959822    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-proxy-tls\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:50.965561 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:50.965538    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpnst\" (UniqueName: \"kubernetes.io/projected/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-api-access-lpnst\") pod \"isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") " pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:51.075466 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:51.075371    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:51.209689 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:51.209653    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"]
Jul 21 18:33:51.212855 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:33:51.212823    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25eee1eb_3045_4b90_b9e1_6e2fca1fa167.slice/crio-47457fbbb5d3b0b262dc8c60639a429400555f4cecd97c1db70a6956f9fd94b8 WatchSource:0}: Error finding container 47457fbbb5d3b0b262dc8c60639a429400555f4cecd97c1db70a6956f9fd94b8: Status 404 returned error can't find the container with id 47457fbbb5d3b0b262dc8c60639a429400555f4cecd97c1db70a6956f9fd94b8
Jul 21 18:33:51.518087 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:51.518045    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" event={"ID":"25eee1eb-3045-4b90-b9e1-6e2fca1fa167","Type":"ContainerStarted","Data":"95b0faa86a6ecc27221c7325feebe00e24ebdc1fa30be6f81037c8fb156cea6c"}
Jul 21 18:33:51.518286 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:51.518122    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" event={"ID":"25eee1eb-3045-4b90-b9e1-6e2fca1fa167","Type":"ContainerStarted","Data":"47457fbbb5d3b0b262dc8c60639a429400555f4cecd97c1db70a6956f9fd94b8"}
Jul 21 18:33:52.367316 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:52.367268    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 21 18:33:55.538406 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:55.538371    2563 generic.go:356] "Generic (PLEG): container finished" podID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerID="95b0faa86a6ecc27221c7325feebe00e24ebdc1fa30be6f81037c8fb156cea6c" exitCode=0
Jul 21 18:33:55.538695 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:55.538441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" event={"ID":"25eee1eb-3045-4b90-b9e1-6e2fca1fa167","Type":"ContainerDied","Data":"95b0faa86a6ecc27221c7325feebe00e24ebdc1fa30be6f81037c8fb156cea6c"}
Jul 21 18:33:56.545373 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:56.545338    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" event={"ID":"25eee1eb-3045-4b90-b9e1-6e2fca1fa167","Type":"ContainerStarted","Data":"9216b6db1cb349c053beecfdba3dbd6253db0571e3644cb80d6339feed092192"}
Jul 21 18:33:56.545763 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:56.545400    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" event={"ID":"25eee1eb-3045-4b90-b9e1-6e2fca1fa167","Type":"ContainerStarted","Data":"f646123ce4c38a0bb69e18305b748d925b2267efec3f2e667e4cf2690959be1d"}
Jul 21 18:33:56.545763 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:56.545599    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:56.567609 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:56.567562    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podStartSLOduration=2.327076402 podStartE2EDuration="6.567542281s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.240465879s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:33:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:33:56.565454131 +0000 UTC m=+4096.123461253" watchObservedRunningTime="2026-07-21 18:33:56.567542281 +0000 UTC m=+4096.125549388"
Jul 21 18:33:57.367019 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:57.366971    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 21 18:33:57.367226 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:57.367111    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:33:57.550162 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:57.550130    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:33:57.551291 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:57.551264    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 18:33:58.555123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:33:58.555080    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 18:34:02.366971 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:02.366928    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 21 18:34:03.559335 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:03.559303    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:34:03.559810 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:03.559759    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 21 18:34:07.367141 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:07.367101    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 21 18:34:12.367373 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:12.367325    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.40:8643/healthz\": dial tcp 10.133.0.40:8643: connect: connection refused"
Jul 21 18:34:13.560784 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:13.560753    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:34:16.234197 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.234168    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:34:16.379943 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.379854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pttw2\" (UniqueName: \"kubernetes.io/projected/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-api-access-pttw2\") pod \"b02bcdad-f065-48d2-8537-fb491e25b93f\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") "
Jul 21 18:34:16.379943 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.379906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-rbac-proxy-sar-config\") pod \"b02bcdad-f065-48d2-8537-fb491e25b93f\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") "
Jul 21 18:34:16.380145 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.380037    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls\") pod \"b02bcdad-f065-48d2-8537-fb491e25b93f\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") "
Jul 21 18:34:16.380145 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.380075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02bcdad-f065-48d2-8537-fb491e25b93f-kserve-provision-location\") pod \"b02bcdad-f065-48d2-8537-fb491e25b93f\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") "
Jul 21 18:34:16.380249 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.380209    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b02bcdad-f065-48d2-8537-fb491e25b93f" (UID: "b02bcdad-f065-48d2-8537-fb491e25b93f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:34:16.380249 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.380222    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-cabundle-cert\") pod \"b02bcdad-f065-48d2-8537-fb491e25b93f\" (UID: \"b02bcdad-f065-48d2-8537-fb491e25b93f\") "
Jul 21 18:34:16.380535 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.380506    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b02bcdad-f065-48d2-8537-fb491e25b93f" (UID: "b02bcdad-f065-48d2-8537-fb491e25b93f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:34:16.380535 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.380521    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:34:16.382147 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.382120    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-api-access-pttw2" (OuterVolumeSpecName: "kube-api-access-pttw2") pod "b02bcdad-f065-48d2-8537-fb491e25b93f" (UID: "b02bcdad-f065-48d2-8537-fb491e25b93f"). InnerVolumeSpecName "kube-api-access-pttw2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:34:16.382245 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.382167    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b02bcdad-f065-48d2-8537-fb491e25b93f" (UID: "b02bcdad-f065-48d2-8537-fb491e25b93f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:34:16.391163 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.391141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b02bcdad-f065-48d2-8537-fb491e25b93f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b02bcdad-f065-48d2-8537-fb491e25b93f" (UID: "b02bcdad-f065-48d2-8537-fb491e25b93f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:34:16.481258 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.481225    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pttw2\" (UniqueName: \"kubernetes.io/projected/b02bcdad-f065-48d2-8537-fb491e25b93f-kube-api-access-pttw2\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:34:16.481258 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.481250    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b02bcdad-f065-48d2-8537-fb491e25b93f-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:34:16.481258 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.481260    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b02bcdad-f065-48d2-8537-fb491e25b93f-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:34:16.481258 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.481268    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b02bcdad-f065-48d2-8537-fb491e25b93f-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:34:16.645501 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.645413    2563 generic.go:356] "Generic (PLEG): container finished" podID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerID="7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586" exitCode=137
Jul 21 18:34:16.645501 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.645476    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" event={"ID":"b02bcdad-f065-48d2-8537-fb491e25b93f","Type":"ContainerDied","Data":"7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586"}
Jul 21 18:34:16.645501 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.645492    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"
Jul 21 18:34:16.645707 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.645565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt" event={"ID":"b02bcdad-f065-48d2-8537-fb491e25b93f","Type":"ContainerDied","Data":"d705f5dbef059ed8971e7a7b2665f8cd7099cc960b2d80c8dc66a73fca73e3a4"}
Jul 21 18:34:16.645707 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.645604    2563 scope.go:117] "RemoveContainer" containerID="8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86"
Jul 21 18:34:16.654472 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.654455    2563 scope.go:117] "RemoveContainer" containerID="7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586"
Jul 21 18:34:16.661905 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.661887    2563 scope.go:117] "RemoveContainer" containerID="bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd"
Jul 21 18:34:16.667397 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.667375    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"]
Jul 21 18:34:16.670662 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.670639    2563 scope.go:117] "RemoveContainer" containerID="8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86"
Jul 21 18:34:16.671007 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:34:16.670972    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86\": container with ID starting with 8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86 not found: ID does not exist" containerID="8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86"
Jul 21 18:34:16.671086 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.671010    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86"} err="failed to get container status \"8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86\": rpc error: code = NotFound desc = could not find container \"8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86\": container with ID starting with 8513bdb5c483f5a0116da61efa8562c8c8702f4869882bfe4780527390157b86 not found: ID does not exist"
Jul 21 18:34:16.671086 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.671031    2563 scope.go:117] "RemoveContainer" containerID="7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586"
Jul 21 18:34:16.671207 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.671121    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-predictor-547dfc5669-9hbtt"]
Jul 21 18:34:16.671326 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:34:16.671305    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586\": container with ID starting with 7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586 not found: ID does not exist" containerID="7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586"
Jul 21 18:34:16.671367 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.671335    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586"} err="failed to get container status \"7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586\": rpc error: code = NotFound desc = could not find container \"7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586\": container with ID starting with 7e4f75a403c369fe8afb12fd54eaf12ed999a0768633b7fb161b46c4af622586 not found: ID does not exist"
Jul 21 18:34:16.671367 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.671351    2563 scope.go:117] "RemoveContainer" containerID="bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd"
Jul 21 18:34:16.671596 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:34:16.671580    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd\": container with ID starting with bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd not found: ID does not exist" containerID="bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd"
Jul 21 18:34:16.671638 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.671602    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd"} err="failed to get container status \"bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd\": rpc error: code = NotFound desc = could not find container \"bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd\": container with ID starting with bb279352780c01a956a4906e1268973f5742f0516b73364030d437f47ee41dcd not found: ID does not exist"
Jul 21 18:34:16.908184 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:16.908110    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" path="/var/lib/kubelet/pods/b02bcdad-f065-48d2-8537-fb491e25b93f/volumes"
Jul 21 18:34:31.682759 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:31.682661    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"]
Jul 21 18:34:31.683195 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:31.683005    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kserve-container" containerID="cri-o://f646123ce4c38a0bb69e18305b748d925b2267efec3f2e667e4cf2690959be1d" gracePeriod=30
Jul 21 18:34:31.683390 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:31.683367    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy" containerID="cri-o://9216b6db1cb349c053beecfdba3dbd6253db0571e3644cb80d6339feed092192" gracePeriod=30
Jul 21 18:34:32.720854 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:32.720819    2563 generic.go:356] "Generic (PLEG): container finished" podID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerID="9216b6db1cb349c053beecfdba3dbd6253db0571e3644cb80d6339feed092192" exitCode=2
Jul 21 18:34:32.721221 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:32.720856    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" event={"ID":"25eee1eb-3045-4b90-b9e1-6e2fca1fa167","Type":"ContainerDied","Data":"9216b6db1cb349c053beecfdba3dbd6253db0571e3644cb80d6339feed092192"}
Jul 21 18:34:33.555945 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:33.555905    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 21 18:34:38.555567 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:38.555526    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 21 18:34:43.555969 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:43.555911    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 21 18:34:43.556355 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:43.556033    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:34:48.555772 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:48.555726    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 21 18:34:53.556204 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:53.556160    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 21 18:34:58.556042 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:34:58.556005    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 21 18:35:01.846781 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:01.846751    2563 generic.go:356] "Generic (PLEG): container finished" podID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerID="f646123ce4c38a0bb69e18305b748d925b2267efec3f2e667e4cf2690959be1d" exitCode=137
Jul 21 18:35:01.847111 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:01.846829    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" event={"ID":"25eee1eb-3045-4b90-b9e1-6e2fca1fa167","Type":"ContainerDied","Data":"f646123ce4c38a0bb69e18305b748d925b2267efec3f2e667e4cf2690959be1d"}
Jul 21 18:35:02.328707 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.328685    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:35:02.459310 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.459232    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-rbac-proxy-sar-config\") pod \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") "
Jul 21 18:35:02.459310 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.459298    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kserve-provision-location\") pod \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") "
Jul 21 18:35:02.459541 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.459327    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-proxy-tls\") pod \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") "
Jul 21 18:35:02.459541 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.459352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lpnst\" (UniqueName: \"kubernetes.io/projected/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-api-access-lpnst\") pod \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") "
Jul 21 18:35:02.459541 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.459384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-cabundle-cert\") pod \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\" (UID: \"25eee1eb-3045-4b90-b9e1-6e2fca1fa167\") "
Jul 21 18:35:02.459753 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.459725    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25eee1eb-3045-4b90-b9e1-6e2fca1fa167" (UID: "25eee1eb-3045-4b90-b9e1-6e2fca1fa167"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:35:02.459867 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.459844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "25eee1eb-3045-4b90-b9e1-6e2fca1fa167" (UID: "25eee1eb-3045-4b90-b9e1-6e2fca1fa167"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:35:02.461698 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.461660    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-api-access-lpnst" (OuterVolumeSpecName: "kube-api-access-lpnst") pod "25eee1eb-3045-4b90-b9e1-6e2fca1fa167" (UID: "25eee1eb-3045-4b90-b9e1-6e2fca1fa167"). InnerVolumeSpecName "kube-api-access-lpnst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:35:02.461698 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.461686    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25eee1eb-3045-4b90-b9e1-6e2fca1fa167" (UID: "25eee1eb-3045-4b90-b9e1-6e2fca1fa167"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:35:02.468546 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.468525    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "25eee1eb-3045-4b90-b9e1-6e2fca1fa167" (UID: "25eee1eb-3045-4b90-b9e1-6e2fca1fa167"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:35:02.560778 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.560748    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:35:02.560778 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.560776    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpnst\" (UniqueName: \"kubernetes.io/projected/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-api-access-lpnst\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:35:02.560951 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.560806    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:35:02.560951 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.560816    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:35:02.560951 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.560824    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/25eee1eb-3045-4b90-b9e1-6e2fca1fa167-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:35:02.852700 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.852671    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq" event={"ID":"25eee1eb-3045-4b90-b9e1-6e2fca1fa167","Type":"ContainerDied","Data":"47457fbbb5d3b0b262dc8c60639a429400555f4cecd97c1db70a6956f9fd94b8"}
Jul 21 18:35:02.853102 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.852721    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"
Jul 21 18:35:02.853102 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.852739    2563 scope.go:117] "RemoveContainer" containerID="9216b6db1cb349c053beecfdba3dbd6253db0571e3644cb80d6339feed092192"
Jul 21 18:35:02.861634 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.861616    2563 scope.go:117] "RemoveContainer" containerID="f646123ce4c38a0bb69e18305b748d925b2267efec3f2e667e4cf2690959be1d"
Jul 21 18:35:02.869665 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.869647    2563 scope.go:117] "RemoveContainer" containerID="95b0faa86a6ecc27221c7325feebe00e24ebdc1fa30be6f81037c8fb156cea6c"
Jul 21 18:35:02.874017 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.873996    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"]
Jul 21 18:35:02.877917 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.877897    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-tensorflow-runtime-predictor-855cfcd677-ktsrq"]
Jul 21 18:35:02.907707 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:02.907686    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" path="/var/lib/kubelet/pods/25eee1eb-3045-4b90-b9e1-6e2fca1fa167/volumes"
Jul 21 18:35:43.050007 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:43.049979    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:35:43.054114 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:35:43.054092    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:36:54.240866 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.240830    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"]
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241173    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241185    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241195    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="storage-initializer"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241201    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="storage-initializer"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241209    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kserve-container"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241216    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kserve-container"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241221    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241226    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241238    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="storage-initializer"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241244    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="storage-initializer"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241250    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kserve-container"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241255    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kserve-container"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241298    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kube-rbac-proxy"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241306    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kube-rbac-proxy"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241312    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="25eee1eb-3045-4b90-b9e1-6e2fca1fa167" containerName="kserve-container"
Jul 21 18:36:54.241330 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.241320    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b02bcdad-f065-48d2-8537-fb491e25b93f" containerName="kserve-container"
Jul 21 18:36:54.244624 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.244608    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.247746 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.247699    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-predictor-serving-cert\""
Jul 21 18:36:54.247909 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.247745    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:36:54.247909 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.247758    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:36:54.247909 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.247717    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-kube-rbac-proxy-sar-config\""
Jul 21 18:36:54.247909 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.247716    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:36:54.247909 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.247864    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:36:54.248188 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.247979    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:36:54.254895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.254870    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"]
Jul 21 18:36:54.330085 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.330047    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.330085 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.330093    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.330261 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.330150    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7821013f-a2d1-42e1-ae57-9c86c75a054d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.330261 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.330206    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvw84\" (UniqueName: \"kubernetes.io/projected/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-api-access-fvw84\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.330261 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.330249    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7821013f-a2d1-42e1-ae57-9c86c75a054d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.430829 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.430777    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7821013f-a2d1-42e1-ae57-9c86c75a054d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.430927 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.430849    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.430927 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.430870    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.430927 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.430905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7821013f-a2d1-42e1-ae57-9c86c75a054d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.431078 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.430930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvw84\" (UniqueName: \"kubernetes.io/projected/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-api-access-fvw84\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.431247 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.431225    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7821013f-a2d1-42e1-ae57-9c86c75a054d-kserve-provision-location\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.431501 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.431481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.431607 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.431548    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-cabundle-cert\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.433439 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.433416    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7821013f-a2d1-42e1-ae57-9c86c75a054d-proxy-tls\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.438848 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.438827    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvw84\" (UniqueName: \"kubernetes.io/projected/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-api-access-fvw84\") pod \"isvc-xgboost-predictor-7566d65bf7-zczzw\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") " pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.555102 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.555067    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:36:54.684971 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:54.684945    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"]
Jul 21 18:36:54.687061 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:36:54.687024    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7821013f_a2d1_42e1_ae57_9c86c75a054d.slice/crio-f83dd3f96711deba3a77e37ad2a4057028673b0a3a1e7e6210354069c0eee901 WatchSource:0}: Error finding container f83dd3f96711deba3a77e37ad2a4057028673b0a3a1e7e6210354069c0eee901: Status 404 returned error can't find the container with id f83dd3f96711deba3a77e37ad2a4057028673b0a3a1e7e6210354069c0eee901
Jul 21 18:36:55.310591 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:55.310557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" event={"ID":"7821013f-a2d1-42e1-ae57-9c86c75a054d","Type":"ContainerStarted","Data":"770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a"}
Jul 21 18:36:55.310981 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:55.310625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" event={"ID":"7821013f-a2d1-42e1-ae57-9c86c75a054d","Type":"ContainerStarted","Data":"f83dd3f96711deba3a77e37ad2a4057028673b0a3a1e7e6210354069c0eee901"}
Jul 21 18:36:59.327096 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:59.327063    2563 generic.go:356] "Generic (PLEG): container finished" podID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerID="770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a" exitCode=0
Jul 21 18:36:59.327532 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:36:59.327145    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" event={"ID":"7821013f-a2d1-42e1-ae57-9c86c75a054d","Type":"ContainerDied","Data":"770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a"}
Jul 21 18:37:20.429487 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:20.429451    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" event={"ID":"7821013f-a2d1-42e1-ae57-9c86c75a054d","Type":"ContainerStarted","Data":"9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753"}
Jul 21 18:37:20.429933 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:20.429515    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" event={"ID":"7821013f-a2d1-42e1-ae57-9c86c75a054d","Type":"ContainerStarted","Data":"4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88"}
Jul 21 18:37:20.429933 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:20.429819    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:37:20.430024 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:20.429970    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:37:20.431324 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:20.431297    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 18:37:20.450677 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:20.450626    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podStartSLOduration=1.954820956 podStartE2EDuration="26.450611952s" totalImagesPullingTime="20.583063568s" totalInitContainerRuntime="3.912727428s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:36:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:37:20.448555351 +0000 UTC m=+4300.006562458" watchObservedRunningTime="2026-07-21 18:37:20.450611952 +0000 UTC m=+4300.008619058"
Jul 21 18:37:21.434598 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:21.434561    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 18:37:26.439001 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:26.438967    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:37:26.439503 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:26.439418    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 18:37:36.440162 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:36.440074    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 18:37:46.439693 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:46.439653    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 18:37:56.439691 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:37:56.439650    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 18:38:06.440289 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:06.440250    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 18:38:16.440623 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:16.440591    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:38:24.363099 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:24.363063    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"]
Jul 21 18:38:24.363574 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:24.363357    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" containerID="cri-o://4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88" gracePeriod=30
Jul 21 18:38:24.363660 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:24.363423    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kube-rbac-proxy" containerID="cri-o://9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753" gracePeriod=30
Jul 21 18:38:24.698640 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:24.698553    2563 generic.go:356] "Generic (PLEG): container finished" podID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerID="9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753" exitCode=2
Jul 21 18:38:24.698640 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:24.698611    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" event={"ID":"7821013f-a2d1-42e1-ae57-9c86c75a054d","Type":"ContainerDied","Data":"9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753"}
Jul 21 18:38:26.435202 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:26.435159    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.42:8643/healthz\": dial tcp 10.133.0.42:8643: connect: connection refused"
Jul 21 18:38:26.439457 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:26.439424    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 21 18:38:27.599727 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.599705    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:38:27.711553 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.711481    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-rbac-proxy-sar-config\") pod \"7821013f-a2d1-42e1-ae57-9c86c75a054d\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") "
Jul 21 18:38:27.711718 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.711568    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fvw84\" (UniqueName: \"kubernetes.io/projected/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-api-access-fvw84\") pod \"7821013f-a2d1-42e1-ae57-9c86c75a054d\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") "
Jul 21 18:38:27.711718 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.711609    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-cabundle-cert\") pod \"7821013f-a2d1-42e1-ae57-9c86c75a054d\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") "
Jul 21 18:38:27.711718 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.711631    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7821013f-a2d1-42e1-ae57-9c86c75a054d-proxy-tls\") pod \"7821013f-a2d1-42e1-ae57-9c86c75a054d\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") "
Jul 21 18:38:27.711718 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.711657    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7821013f-a2d1-42e1-ae57-9c86c75a054d-kserve-provision-location\") pod \"7821013f-a2d1-42e1-ae57-9c86c75a054d\" (UID: \"7821013f-a2d1-42e1-ae57-9c86c75a054d\") "
Jul 21 18:38:27.711993 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.711970    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7821013f-a2d1-42e1-ae57-9c86c75a054d" (UID: "7821013f-a2d1-42e1-ae57-9c86c75a054d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:38:27.712052 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.712030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7821013f-a2d1-42e1-ae57-9c86c75a054d" (UID: "7821013f-a2d1-42e1-ae57-9c86c75a054d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:38:27.712097 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.712030    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7821013f-a2d1-42e1-ae57-9c86c75a054d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7821013f-a2d1-42e1-ae57-9c86c75a054d" (UID: "7821013f-a2d1-42e1-ae57-9c86c75a054d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:38:27.712753 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.712713    2563 generic.go:356] "Generic (PLEG): container finished" podID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerID="4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88" exitCode=0
Jul 21 18:38:27.712868 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.712767    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"
Jul 21 18:38:27.712914 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.712764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" event={"ID":"7821013f-a2d1-42e1-ae57-9c86c75a054d","Type":"ContainerDied","Data":"4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88"}
Jul 21 18:38:27.712958 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.712941    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw" event={"ID":"7821013f-a2d1-42e1-ae57-9c86c75a054d","Type":"ContainerDied","Data":"f83dd3f96711deba3a77e37ad2a4057028673b0a3a1e7e6210354069c0eee901"}
Jul 21 18:38:27.713008 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.712982    2563 scope.go:117] "RemoveContainer" containerID="9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753"
Jul 21 18:38:27.714120 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.714057    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-api-access-fvw84" (OuterVolumeSpecName: "kube-api-access-fvw84") pod "7821013f-a2d1-42e1-ae57-9c86c75a054d" (UID: "7821013f-a2d1-42e1-ae57-9c86c75a054d"). InnerVolumeSpecName "kube-api-access-fvw84". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:38:27.714120 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.714105    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7821013f-a2d1-42e1-ae57-9c86c75a054d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7821013f-a2d1-42e1-ae57-9c86c75a054d" (UID: "7821013f-a2d1-42e1-ae57-9c86c75a054d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:38:27.728768 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.728748    2563 scope.go:117] "RemoveContainer" containerID="4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88"
Jul 21 18:38:27.736017 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.736002    2563 scope.go:117] "RemoveContainer" containerID="770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a"
Jul 21 18:38:27.743189 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.743173    2563 scope.go:117] "RemoveContainer" containerID="9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753"
Jul 21 18:38:27.743439 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:38:27.743420    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753\": container with ID starting with 9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753 not found: ID does not exist" containerID="9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753"
Jul 21 18:38:27.743487 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.743446    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753"} err="failed to get container status \"9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753\": rpc error: code = NotFound desc = could not find container \"9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753\": container with ID starting with 9c5bdb3957d780ca235dea3523f0c20416196231e535ad827fc82fb2db6a1753 not found: ID does not exist"
Jul 21 18:38:27.743487 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.743462    2563 scope.go:117] "RemoveContainer" containerID="4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88"
Jul 21 18:38:27.743672 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:38:27.743657    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88\": container with ID starting with 4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88 not found: ID does not exist" containerID="4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88"
Jul 21 18:38:27.743713 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.743676    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88"} err="failed to get container status \"4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88\": rpc error: code = NotFound desc = could not find container \"4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88\": container with ID starting with 4476da528fcfd1e49771bbc13fe4d149cd7285bcac5c0605d166ae6a8d104f88 not found: ID does not exist"
Jul 21 18:38:27.743713 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.743689    2563 scope.go:117] "RemoveContainer" containerID="770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a"
Jul 21 18:38:27.743913 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:38:27.743895    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a\": container with ID starting with 770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a not found: ID does not exist" containerID="770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a"
Jul 21 18:38:27.743995 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.743919    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a"} err="failed to get container status \"770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a\": rpc error: code = NotFound desc = could not find container \"770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a\": container with ID starting with 770f6501fb8a1d138c009ed818a1b0745c1f6207707e991c485b2b9d0eddd83a not found: ID does not exist"
Jul 21 18:38:27.812312 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.812287    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvw84\" (UniqueName: \"kubernetes.io/projected/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-api-access-fvw84\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:38:27.812312 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.812309    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:38:27.812423 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.812320    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7821013f-a2d1-42e1-ae57-9c86c75a054d-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:38:27.812423 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.812327    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7821013f-a2d1-42e1-ae57-9c86c75a054d-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:38:27.812423 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:27.812336    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7821013f-a2d1-42e1-ae57-9c86c75a054d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:38:28.034093 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:28.034068    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"]
Jul 21 18:38:28.037692 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:28.037670    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-predictor-7566d65bf7-zczzw"]
Jul 21 18:38:28.907684 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:38:28.907653    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" path="/var/lib/kubelet/pods/7821013f-a2d1-42e1-ae57-9c86c75a054d/volumes"
Jul 21 18:40:20.053235 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.053196    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"]
Jul 21 18:40:20.054098 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.054071    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container"
Jul 21 18:40:20.054269 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.054256    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container"
Jul 21 18:40:20.054386 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.054375    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="storage-initializer"
Jul 21 18:40:20.054466 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.054457    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="storage-initializer"
Jul 21 18:40:20.054574 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.054565    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kube-rbac-proxy"
Jul 21 18:40:20.054652 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.054644    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kube-rbac-proxy"
Jul 21 18:40:20.054947 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.054931    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kserve-container"
Jul 21 18:40:20.055071 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.055059    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7821013f-a2d1-42e1-ae57-9c86c75a054d" containerName="kube-rbac-proxy"
Jul 21 18:40:20.058408 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.058388    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.062030 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.062006    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:40:20.062246 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.062225    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:40:20.063293 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.063271    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:40:20.063412 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.063298    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-predictor-serving-cert\""
Jul 21 18:40:20.063412 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.063313    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-runtime-kube-rbac-proxy-sar-config\""
Jul 21 18:40:20.063412 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.063325    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:40:20.063412 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.063365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:40:20.068048 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.068029    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"]
Jul 21 18:40:20.140017 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.139979    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.140017 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.140016    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce896868-1c07-48fa-afaf-226dff488eba-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.140240 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.140037    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4jwj\" (UniqueName: \"kubernetes.io/projected/ce896868-1c07-48fa-afaf-226dff488eba-kube-api-access-d4jwj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.140240 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.140054    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.140240 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.140107    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce896868-1c07-48fa-afaf-226dff488eba-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.241060 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.241018    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce896868-1c07-48fa-afaf-226dff488eba-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.241236 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.241121    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.241236 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.241143    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce896868-1c07-48fa-afaf-226dff488eba-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.241236 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.241160    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4jwj\" (UniqueName: \"kubernetes.io/projected/ce896868-1c07-48fa-afaf-226dff488eba-kube-api-access-d4jwj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.241236 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.241181    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.241443 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.241409    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce896868-1c07-48fa-afaf-226dff488eba-kserve-provision-location\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.241844 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.241828    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-cabundle-cert\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.241906 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.241838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.243709 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.243688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce896868-1c07-48fa-afaf-226dff488eba-proxy-tls\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.250273 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.250243    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4jwj\" (UniqueName: \"kubernetes.io/projected/ce896868-1c07-48fa-afaf-226dff488eba-kube-api-access-d4jwj\") pod \"isvc-xgboost-runtime-predictor-54dd488c85-hcj22\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") " pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.371354 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.371270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:20.499753 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:20.499687    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"]
Jul 21 18:40:20.502013 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:40:20.501980    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce896868_1c07_48fa_afaf_226dff488eba.slice/crio-3cc7075410774d1a1a80a685a7f93dba0c9dfe9b498d2c10682eca3c0fc5bebe WatchSource:0}: Error finding container 3cc7075410774d1a1a80a685a7f93dba0c9dfe9b498d2c10682eca3c0fc5bebe: Status 404 returned error can't find the container with id 3cc7075410774d1a1a80a685a7f93dba0c9dfe9b498d2c10682eca3c0fc5bebe
Jul 21 18:40:21.187623 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:21.187590    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" event={"ID":"ce896868-1c07-48fa-afaf-226dff488eba","Type":"ContainerStarted","Data":"88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f"}
Jul 21 18:40:21.188065 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:21.187646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" event={"ID":"ce896868-1c07-48fa-afaf-226dff488eba","Type":"ContainerStarted","Data":"3cc7075410774d1a1a80a685a7f93dba0c9dfe9b498d2c10682eca3c0fc5bebe"}
Jul 21 18:40:24.200856 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:24.200822    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce896868-1c07-48fa-afaf-226dff488eba" containerID="88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f" exitCode=0
Jul 21 18:40:24.201249 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:24.200880    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" event={"ID":"ce896868-1c07-48fa-afaf-226dff488eba","Type":"ContainerDied","Data":"88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f"}
Jul 21 18:40:25.207679 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:25.207643    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" event={"ID":"ce896868-1c07-48fa-afaf-226dff488eba","Type":"ContainerStarted","Data":"27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93"}
Jul 21 18:40:25.208127 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:25.207715    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" event={"ID":"ce896868-1c07-48fa-afaf-226dff488eba","Type":"ContainerStarted","Data":"9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9"}
Jul 21 18:40:25.208127 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:25.208043    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:25.227725 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:25.227668    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podStartSLOduration=1.720956289 podStartE2EDuration="5.227652733s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.506696444s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:40:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:40:25.225890032 +0000 UTC m=+4484.783897141" watchObservedRunningTime="2026-07-21 18:40:25.227652733 +0000 UTC m=+4484.785659839"
Jul 21 18:40:26.212496 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:26.212458    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:26.213710 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:26.213679    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:40:27.216415 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:27.216374    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:40:32.220919 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:32.220845    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:40:32.221402 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:32.221373    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:40:42.222014 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:42.221977    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:40:43.079215 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:43.079189    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:40:43.083557 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:43.083532    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:40:52.221370 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:40:52.221329    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:41:02.222002 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:02.221961    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:41:12.221370 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:12.221332    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:41:22.221733 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:22.221695    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:41:32.222598 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:32.222563    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:41:40.190312 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:40.190275    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"]
Jul 21 18:41:40.190927 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:40.190597    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" containerID="cri-o://9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9" gracePeriod=30
Jul 21 18:41:40.190927 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:40.190640    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kube-rbac-proxy" containerID="cri-o://27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93" gracePeriod=30
Jul 21 18:41:40.544188 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:40.544155    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce896868-1c07-48fa-afaf-226dff488eba" containerID="27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93" exitCode=2
Jul 21 18:41:40.544386 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:40.544234    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" event={"ID":"ce896868-1c07-48fa-afaf-226dff488eba","Type":"ContainerDied","Data":"27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93"}
Jul 21 18:41:42.217490 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:42.217442    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 21 18:41:42.221839 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:42.221810    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 21 18:41:43.836615 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.836593    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:41:43.959719 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.959629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d4jwj\" (UniqueName: \"kubernetes.io/projected/ce896868-1c07-48fa-afaf-226dff488eba-kube-api-access-d4jwj\") pod \"ce896868-1c07-48fa-afaf-226dff488eba\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") "
Jul 21 18:41:43.959719 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.959676    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce896868-1c07-48fa-afaf-226dff488eba-proxy-tls\") pod \"ce896868-1c07-48fa-afaf-226dff488eba\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") "
Jul 21 18:41:43.959719 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.959700    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce896868-1c07-48fa-afaf-226dff488eba-kserve-provision-location\") pod \"ce896868-1c07-48fa-afaf-226dff488eba\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") "
Jul 21 18:41:43.959995 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.959743    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-kube-rbac-proxy-sar-config\") pod \"ce896868-1c07-48fa-afaf-226dff488eba\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") "
Jul 21 18:41:43.959995 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.959809    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-cabundle-cert\") pod \"ce896868-1c07-48fa-afaf-226dff488eba\" (UID: \"ce896868-1c07-48fa-afaf-226dff488eba\") "
Jul 21 18:41:43.960101 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.960064    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ce896868-1c07-48fa-afaf-226dff488eba-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "ce896868-1c07-48fa-afaf-226dff488eba" (UID: "ce896868-1c07-48fa-afaf-226dff488eba"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:41:43.960187 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.960127    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ce896868-1c07-48fa-afaf-226dff488eba" (UID: "ce896868-1c07-48fa-afaf-226dff488eba"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:41:43.960316 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.960297    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "ce896868-1c07-48fa-afaf-226dff488eba" (UID: "ce896868-1c07-48fa-afaf-226dff488eba"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:41:43.962014 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.961986    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ce896868-1c07-48fa-afaf-226dff488eba-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ce896868-1c07-48fa-afaf-226dff488eba" (UID: "ce896868-1c07-48fa-afaf-226dff488eba"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:41:43.962125 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:43.962020    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce896868-1c07-48fa-afaf-226dff488eba-kube-api-access-d4jwj" (OuterVolumeSpecName: "kube-api-access-d4jwj") pod "ce896868-1c07-48fa-afaf-226dff488eba" (UID: "ce896868-1c07-48fa-afaf-226dff488eba"). InnerVolumeSpecName "kube-api-access-d4jwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:41:44.060646 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.060603    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:41:44.060646 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.060639    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/ce896868-1c07-48fa-afaf-226dff488eba-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:41:44.060646 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.060652    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4jwj\" (UniqueName: \"kubernetes.io/projected/ce896868-1c07-48fa-afaf-226dff488eba-kube-api-access-d4jwj\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:41:44.060920 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.060664    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ce896868-1c07-48fa-afaf-226dff488eba-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:41:44.060920 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.060677    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/ce896868-1c07-48fa-afaf-226dff488eba-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:41:44.563042 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.563005    2563 generic.go:356] "Generic (PLEG): container finished" podID="ce896868-1c07-48fa-afaf-226dff488eba" containerID="9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9" exitCode=0
Jul 21 18:41:44.563256 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.563077    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"
Jul 21 18:41:44.563256 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.563087    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" event={"ID":"ce896868-1c07-48fa-afaf-226dff488eba","Type":"ContainerDied","Data":"9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9"}
Jul 21 18:41:44.563256 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.563151    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22" event={"ID":"ce896868-1c07-48fa-afaf-226dff488eba","Type":"ContainerDied","Data":"3cc7075410774d1a1a80a685a7f93dba0c9dfe9b498d2c10682eca3c0fc5bebe"}
Jul 21 18:41:44.563256 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.563179    2563 scope.go:117] "RemoveContainer" containerID="27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93"
Jul 21 18:41:44.572300 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.572283    2563 scope.go:117] "RemoveContainer" containerID="9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9"
Jul 21 18:41:44.579850 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.579823    2563 scope.go:117] "RemoveContainer" containerID="88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f"
Jul 21 18:41:44.587725 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.587698    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"]
Jul 21 18:41:44.588479 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.588460    2563 scope.go:117] "RemoveContainer" containerID="27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93"
Jul 21 18:41:44.589013 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:41:44.588990    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93\": container with ID starting with 27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93 not found: ID does not exist" containerID="27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93"
Jul 21 18:41:44.589121 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.589024    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93"} err="failed to get container status \"27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93\": rpc error: code = NotFound desc = could not find container \"27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93\": container with ID starting with 27ccf4894ba35b1ea5507f93c76a2cc2ff24f257cb7087b983446177d9c81e93 not found: ID does not exist"
Jul 21 18:41:44.589121 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.589049    2563 scope.go:117] "RemoveContainer" containerID="9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9"
Jul 21 18:41:44.589332 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:41:44.589314    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9\": container with ID starting with 9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9 not found: ID does not exist" containerID="9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9"
Jul 21 18:41:44.589385 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.589338    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9"} err="failed to get container status \"9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9\": rpc error: code = NotFound desc = could not find container \"9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9\": container with ID starting with 9e5ef426e94dc19efc2f13bd667a26e995f1abbfb8639ceb6d0fd5d40ec546d9 not found: ID does not exist"
Jul 21 18:41:44.589385 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.589353    2563 scope.go:117] "RemoveContainer" containerID="88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f"
Jul 21 18:41:44.589604 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:41:44.589588    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f\": container with ID starting with 88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f not found: ID does not exist" containerID="88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f"
Jul 21 18:41:44.589662 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.589612    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f"} err="failed to get container status \"88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f\": rpc error: code = NotFound desc = could not find container \"88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f\": container with ID starting with 88950c42aadef8a9189985b4a3203ff6b4768d9bf72c6ffb09e8333edf948f5f not found: ID does not exist"
Jul 21 18:41:44.590307 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.590288    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-runtime-predictor-54dd488c85-hcj22"]
Jul 21 18:41:44.908899 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:41:44.908818    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce896868-1c07-48fa-afaf-226dff488eba" path="/var/lib/kubelet/pods/ce896868-1c07-48fa-afaf-226dff488eba/volumes"
Jul 21 18:42:50.669164 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669126    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"]
Jul 21 18:42:50.669572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669444    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="storage-initializer"
Jul 21 18:42:50.669572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669455    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="storage-initializer"
Jul 21 18:42:50.669572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669473    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container"
Jul 21 18:42:50.669572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669478    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container"
Jul 21 18:42:50.669572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669485    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kube-rbac-proxy"
Jul 21 18:42:50.669572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669490    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kube-rbac-proxy"
Jul 21 18:42:50.669572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669538    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kserve-container"
Jul 21 18:42:50.669572 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.669550    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce896868-1c07-48fa-afaf-226dff488eba" containerName="kube-rbac-proxy"
Jul 21 18:42:50.673956 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.673940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.676757 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.676732    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 21 18:42:50.676895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.676815    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 21 18:42:50.676895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.676824    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 21 18:42:50.676895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.676871    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-jg28b\""
Jul 21 18:42:50.676895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.676876    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 21 18:42:50.676895 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.676820    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 21 18:42:50.677997 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.677983    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 21 18:42:50.686359 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.686336    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"]
Jul 21 18:42:50.819457 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.819420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.819457 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.819456    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.819657 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.819475    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.819657 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.819498    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.819657 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.819576    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cwt64\" (UniqueName: \"kubernetes.io/projected/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-api-access-cwt64\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.920777 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.920684    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.920777 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.920718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.920777 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.920735    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.920777 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.920757    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.921212 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.920815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cwt64\" (UniqueName: \"kubernetes.io/projected/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-api-access-cwt64\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.921212 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:42:50.920881    2563 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-xgboost-v2-predictor-serving-cert: secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 21 18:42:50.921212 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:42:50.920964    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls podName:0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58 nodeName:}" failed. No retries permitted until 2026-07-21 18:42:51.420935084 +0000 UTC m=+4630.978942188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls") pod "isvc-xgboost-v2-predictor-fcc668bc-b5zhm" (UID: "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58") : secret "isvc-xgboost-v2-predictor-serving-cert" not found
Jul 21 18:42:50.921212 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.921145    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.921513 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.921492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.921582 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.921523    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:50.930049 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:50.930026    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cwt64\" (UniqueName: \"kubernetes.io/projected/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-api-access-cwt64\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:51.425459 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:51.425420    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:51.427908 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:51.427884    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-b5zhm\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:51.584627 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:51.584589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:51.721334 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:51.721309    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"]
Jul 21 18:42:51.723201 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:42:51.723176    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f4a80fb_a21b_4e2f_9c9b_98b4e52b6c58.slice/crio-cec8d3c15fd79529ab878e932a3369aa479b65d183d4b5844e92b9b9f4a36bd4 WatchSource:0}: Error finding container cec8d3c15fd79529ab878e932a3369aa479b65d183d4b5844e92b9b9f4a36bd4: Status 404 returned error can't find the container with id cec8d3c15fd79529ab878e932a3369aa479b65d183d4b5844e92b9b9f4a36bd4
Jul 21 18:42:51.849937 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:51.849900    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" event={"ID":"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58","Type":"ContainerStarted","Data":"9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801"}
Jul 21 18:42:51.850101 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:51.850001    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" event={"ID":"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58","Type":"ContainerStarted","Data":"cec8d3c15fd79529ab878e932a3369aa479b65d183d4b5844e92b9b9f4a36bd4"}
Jul 21 18:42:55.869993 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:55.869945    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerID="9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801" exitCode=0
Jul 21 18:42:55.870412 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:55.870023    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" event={"ID":"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58","Type":"ContainerDied","Data":"9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801"}
Jul 21 18:42:56.876036 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:56.876000    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" event={"ID":"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58","Type":"ContainerStarted","Data":"ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078"}
Jul 21 18:42:56.876477 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:56.876059    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" event={"ID":"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58","Type":"ContainerStarted","Data":"e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a"}
Jul 21 18:42:56.876477 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:56.876343    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:56.876477 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:56.876402    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:42:56.877446 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:56.877418    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 21 18:42:56.895753 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:56.895713    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podStartSLOduration=2.9909461200000003 podStartE2EDuration="6.89569979s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.90475367s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:42:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:42:56.895454023 +0000 UTC m=+4636.453461129" watchObservedRunningTime="2026-07-21 18:42:56.89569979 +0000 UTC m=+4636.453706898"
Jul 21 18:42:57.880626 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:42:57.880585    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 21 18:43:02.884964 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:43:02.884935    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:43:02.885482 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:43:02.885445    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 21 18:43:12.885734 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:43:12.885689    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 21 18:43:22.885867 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:43:22.885823    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 21 18:43:32.885727 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:43:32.885632    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 21 18:43:42.886305 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:43:42.886265    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 21 18:43:52.886137 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:43:52.886108    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:44:00.782243 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:00.782212    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"]
Jul 21 18:44:00.782633 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:00.782418    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" containerID="cri-o://e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a" gracePeriod=30
Jul 21 18:44:00.782633 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:00.782500    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kube-rbac-proxy" containerID="cri-o://ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078" gracePeriod=30
Jul 21 18:44:01.136473 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:01.136399    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerID="ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078" exitCode=2
Jul 21 18:44:01.136616 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:01.136477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" event={"ID":"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58","Type":"ContainerDied","Data":"ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078"}
Jul 21 18:44:02.881391 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:02.881350    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 21 18:44:02.885666 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:02.885639    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 21 18:44:04.131673 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.131650    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:44:04.152053 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.152023    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerID="e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a" exitCode=0
Jul 21 18:44:04.152197 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.152090    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"
Jul 21 18:44:04.152197 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.152085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" event={"ID":"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58","Type":"ContainerDied","Data":"e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a"}
Jul 21 18:44:04.152197 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.152191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm" event={"ID":"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58","Type":"ContainerDied","Data":"cec8d3c15fd79529ab878e932a3369aa479b65d183d4b5844e92b9b9f4a36bd4"}
Jul 21 18:44:04.152341 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.152236    2563 scope.go:117] "RemoveContainer" containerID="ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078"
Jul 21 18:44:04.161227 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.161195    2563 scope.go:117] "RemoveContainer" containerID="e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a"
Jul 21 18:44:04.170171 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.169956    2563 scope.go:117] "RemoveContainer" containerID="9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801"
Jul 21 18:44:04.178560 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.178545    2563 scope.go:117] "RemoveContainer" containerID="ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078"
Jul 21 18:44:04.178839 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:44:04.178819    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078\": container with ID starting with ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078 not found: ID does not exist" containerID="ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078"
Jul 21 18:44:04.178897 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.178850    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078"} err="failed to get container status \"ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078\": rpc error: code = NotFound desc = could not find container \"ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078\": container with ID starting with ae49c01d6cc5bebb9ed2a07d0858538925075c2809742454d77b3a283f903078 not found: ID does not exist"
Jul 21 18:44:04.178897 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.178869    2563 scope.go:117] "RemoveContainer" containerID="e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a"
Jul 21 18:44:04.179162 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:44:04.179143    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a\": container with ID starting with e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a not found: ID does not exist" containerID="e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a"
Jul 21 18:44:04.179218 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.179168    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a"} err="failed to get container status \"e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a\": rpc error: code = NotFound desc = could not find container \"e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a\": container with ID starting with e941d87e2271e1aafec89259067cd9626e4111e0243f4abea96f22c85a52a88a not found: ID does not exist"
Jul 21 18:44:04.179218 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.179185    2563 scope.go:117] "RemoveContainer" containerID="9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801"
Jul 21 18:44:04.179391 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:44:04.179372    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801\": container with ID starting with 9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801 not found: ID does not exist" containerID="9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801"
Jul 21 18:44:04.179434 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.179394    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801"} err="failed to get container status \"9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801\": rpc error: code = NotFound desc = could not find container \"9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801\": container with ID starting with 9ef65fc9ffbbea6d549059ecc2a20ed501f0fb2bd2035070dc6f3bdf4d161801 not found: ID does not exist"
Jul 21 18:44:04.302864 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.302829    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-cabundle-cert\") pod \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") "
Jul 21 18:44:04.303012 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.302919    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cwt64\" (UniqueName: \"kubernetes.io/projected/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-api-access-cwt64\") pod \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") "
Jul 21 18:44:04.303012 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.302972    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-rbac-proxy-sar-config\") pod \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") "
Jul 21 18:44:04.303120 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.303021    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kserve-provision-location\") pod \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") "
Jul 21 18:44:04.303120 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.303055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls\") pod \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\" (UID: \"0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58\") "
Jul 21 18:44:04.303345 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.303215    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" (UID: "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:44:04.303421 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.303353    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" (UID: "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:44:04.303421 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.303372    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" (UID: "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 21 18:44:04.305217 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.305191    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-api-access-cwt64" (OuterVolumeSpecName: "kube-api-access-cwt64") pod "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" (UID: "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58"). InnerVolumeSpecName "kube-api-access-cwt64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:44:04.305217 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.305206    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" (UID: "0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 21 18:44:04.403987 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.403963    2563 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-cabundle-cert\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:44:04.403987 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.403984    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cwt64\" (UniqueName: \"kubernetes.io/projected/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-api-access-cwt64\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:44:04.404123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.403995    2563 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:44:04.404123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.404005    2563 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-kserve-provision-location\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:44:04.404123 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.404014    2563 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58-proxy-tls\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:44:04.474358 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.474330    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"]
Jul 21 18:44:04.477885 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.477863    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-b5zhm"]
Jul 21 18:44:04.908679 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:44:04.908648    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" path="/var/lib/kubelet/pods/0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58/volumes"
Jul 21 18:45:43.106673 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:45:43.106648    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:45:43.112197 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:45:43.112177    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:50:23.541172 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541139    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-27tlh/must-gather-t9blg"]
Jul 21 18:50:23.541619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541449    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="storage-initializer"
Jul 21 18:50:23.541619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541460    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="storage-initializer"
Jul 21 18:50:23.541619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541471    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container"
Jul 21 18:50:23.541619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541477    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container"
Jul 21 18:50:23.541619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541488    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kube-rbac-proxy"
Jul 21 18:50:23.541619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541494    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kube-rbac-proxy"
Jul 21 18:50:23.541619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541565    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kube-rbac-proxy"
Jul 21 18:50:23.541619 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.541575    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f4a80fb-a21b-4e2f-9c9b-98b4e52b6c58" containerName="kserve-container"
Jul 21 18:50:23.544912 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.544894    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:50:23.547630 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.547610    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-27tlh\"/\"kube-root-ca.crt\""
Jul 21 18:50:23.547682 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.547610    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-27tlh\"/\"openshift-service-ca.crt\""
Jul 21 18:50:23.552428 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.552407    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-27tlh/must-gather-t9blg"]
Jul 21 18:50:23.668415 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.668377    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgs9s\" (UniqueName: \"kubernetes.io/projected/7673628b-2e58-4b30-ba38-5eb25b66dd83-kube-api-access-zgs9s\") pod \"must-gather-t9blg\" (UID: \"7673628b-2e58-4b30-ba38-5eb25b66dd83\") " pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:50:23.668611 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.668442    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7673628b-2e58-4b30-ba38-5eb25b66dd83-must-gather-output\") pod \"must-gather-t9blg\" (UID: \"7673628b-2e58-4b30-ba38-5eb25b66dd83\") " pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:50:23.769478 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.769438    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgs9s\" (UniqueName: \"kubernetes.io/projected/7673628b-2e58-4b30-ba38-5eb25b66dd83-kube-api-access-zgs9s\") pod \"must-gather-t9blg\" (UID: \"7673628b-2e58-4b30-ba38-5eb25b66dd83\") " pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:50:23.769667 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.769540    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7673628b-2e58-4b30-ba38-5eb25b66dd83-must-gather-output\") pod \"must-gather-t9blg\" (UID: \"7673628b-2e58-4b30-ba38-5eb25b66dd83\") " pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:50:23.769935 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.769917    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7673628b-2e58-4b30-ba38-5eb25b66dd83-must-gather-output\") pod \"must-gather-t9blg\" (UID: \"7673628b-2e58-4b30-ba38-5eb25b66dd83\") " pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:50:23.778604 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.778568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgs9s\" (UniqueName: \"kubernetes.io/projected/7673628b-2e58-4b30-ba38-5eb25b66dd83-kube-api-access-zgs9s\") pod \"must-gather-t9blg\" (UID: \"7673628b-2e58-4b30-ba38-5eb25b66dd83\") " pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:50:23.862819 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.862707    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:50:23.999202 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:23.999149    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-27tlh/must-gather-t9blg"]
Jul 21 18:50:24.001330 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:50:24.001305    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7673628b_2e58_4b30_ba38_5eb25b66dd83.slice/crio-f8a130009b35a7b2214cb2748b4527e24b8b0f65601af2e13572ac7e090f3a69 WatchSource:0}: Error finding container f8a130009b35a7b2214cb2748b4527e24b8b0f65601af2e13572ac7e090f3a69: Status 404 returned error can't find the container with id f8a130009b35a7b2214cb2748b4527e24b8b0f65601af2e13572ac7e090f3a69
Jul 21 18:50:24.003023 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:24.003008    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 21 18:50:24.698815 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:24.698764    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27tlh/must-gather-t9blg" event={"ID":"7673628b-2e58-4b30-ba38-5eb25b66dd83","Type":"ContainerStarted","Data":"f8a130009b35a7b2214cb2748b4527e24b8b0f65601af2e13572ac7e090f3a69"}
Jul 21 18:50:30.731393 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:30.730698    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27tlh/must-gather-t9blg" event={"ID":"7673628b-2e58-4b30-ba38-5eb25b66dd83","Type":"ContainerStarted","Data":"384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b"}
Jul 21 18:50:30.731393 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:30.730750    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27tlh/must-gather-t9blg" event={"ID":"7673628b-2e58-4b30-ba38-5eb25b66dd83","Type":"ContainerStarted","Data":"96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368"}
Jul 21 18:50:30.749090 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:30.749046    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-27tlh/must-gather-t9blg" podStartSLOduration=1.3646223659999999 podStartE2EDuration="7.749030285s" totalImagesPullingTime="6.384407919s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-21 18:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:50:30.747670857 +0000 UTC m=+5090.305677991" watchObservedRunningTime="2026-07-21 18:50:30.749030285 +0000 UTC m=+5090.307037392"
Jul 21 18:50:43.138042 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:43.138016    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:50:43.148888 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:43.148850    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:50:55.845824 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:55.845773    2563 generic.go:356] "Generic (PLEG): container finished" podID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerID="96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368" exitCode=0
Jul 21 18:50:55.846228 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:55.845848    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-27tlh/must-gather-t9blg" event={"ID":"7673628b-2e58-4b30-ba38-5eb25b66dd83","Type":"ContainerDied","Data":"96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368"}
Jul 21 18:50:55.846228 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:55.846117    2563 scope.go:117] "RemoveContainer" containerID="96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368"
Jul 21 18:50:55.890981 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:55.890959    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-27tlh_must-gather-t9blg_7673628b-2e58-4b30-ba38-5eb25b66dd83/gather/0.log"
Jul 21 18:50:59.349633 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:59.349605    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-p8lvn_13d2df84-0465-42b2-9ff8-cb39aaa95e33/global-pull-secret-syncer/0.log"
Jul 21 18:50:59.550294 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:59.550271    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-v89wr_7573a048-9229-4379-812a-1f7e92b6a852/konnectivity-agent/0.log"
Jul 21 18:50:59.635841 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:50:59.635752    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-22.ec2.internal_c4726a2cf6e4377be6ec06d1c802bab6/haproxy/0.log"
Jul 21 18:51:01.431697 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.431616    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-27tlh/must-gather-t9blg"]
Jul 21 18:51:01.432230 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.431811    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-27tlh/must-gather-t9blg" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerName="copy" containerID="cri-o://384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b" gracePeriod=2
Jul 21 18:51:01.434248 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:51:01.434220    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t9blg\" is forbidden: User \"system:node:ip-10-0-129-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-27tlh\": no relationship found between node 'ip-10-0-129-22.ec2.internal' and this object" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:51:01.436305 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.435933    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-27tlh/must-gather-t9blg"]
Jul 21 18:51:01.666311 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.666287    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-27tlh_must-gather-t9blg_7673628b-2e58-4b30-ba38-5eb25b66dd83/copy/0.log"
Jul 21 18:51:01.667952 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.667933    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:51:01.670234 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:51:01.670213    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t9blg\" is forbidden: User \"system:node:ip-10-0-129-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-27tlh\": no relationship found between node 'ip-10-0-129-22.ec2.internal' and this object" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:51:01.712941 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.712876    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zgs9s\" (UniqueName: \"kubernetes.io/projected/7673628b-2e58-4b30-ba38-5eb25b66dd83-kube-api-access-zgs9s\") pod \"7673628b-2e58-4b30-ba38-5eb25b66dd83\" (UID: \"7673628b-2e58-4b30-ba38-5eb25b66dd83\") "
Jul 21 18:51:01.713055 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.713006    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7673628b-2e58-4b30-ba38-5eb25b66dd83-must-gather-output\") pod \"7673628b-2e58-4b30-ba38-5eb25b66dd83\" (UID: \"7673628b-2e58-4b30-ba38-5eb25b66dd83\") "
Jul 21 18:51:01.714076 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.714053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7673628b-2e58-4b30-ba38-5eb25b66dd83-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "7673628b-2e58-4b30-ba38-5eb25b66dd83" (UID: "7673628b-2e58-4b30-ba38-5eb25b66dd83"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 21 18:51:01.715179 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.715160    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7673628b-2e58-4b30-ba38-5eb25b66dd83-kube-api-access-zgs9s" (OuterVolumeSpecName: "kube-api-access-zgs9s") pod "7673628b-2e58-4b30-ba38-5eb25b66dd83" (UID: "7673628b-2e58-4b30-ba38-5eb25b66dd83"). InnerVolumeSpecName "kube-api-access-zgs9s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 21 18:51:01.814069 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.814040    2563 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/7673628b-2e58-4b30-ba38-5eb25b66dd83-must-gather-output\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:51:01.814069 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.814064    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zgs9s\" (UniqueName: \"kubernetes.io/projected/7673628b-2e58-4b30-ba38-5eb25b66dd83-kube-api-access-zgs9s\") on node \"ip-10-0-129-22.ec2.internal\" DevicePath \"\""
Jul 21 18:51:01.870954 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.870933    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-27tlh_must-gather-t9blg_7673628b-2e58-4b30-ba38-5eb25b66dd83/copy/0.log"
Jul 21 18:51:01.871434 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.871410    2563 generic.go:356] "Generic (PLEG): container finished" podID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerID="384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b" exitCode=143
Jul 21 18:51:01.871530 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.871472    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:51:01.871530 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.871515    2563 scope.go:117] "RemoveContainer" containerID="384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b"
Jul 21 18:51:01.873929 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:51:01.873898    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t9blg\" is forbidden: User \"system:node:ip-10-0-129-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-27tlh\": no relationship found between node 'ip-10-0-129-22.ec2.internal' and this object" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:51:01.880146 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.880131    2563 scope.go:117] "RemoveContainer" containerID="96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368"
Jul 21 18:51:01.882198 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:51:01.882173    2563 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-t9blg\" is forbidden: User \"system:node:ip-10-0-129-22.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-27tlh\": no relationship found between node 'ip-10-0-129-22.ec2.internal' and this object" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" pod="openshift-must-gather-27tlh/must-gather-t9blg"
Jul 21 18:51:01.892048 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.892031    2563 scope.go:117] "RemoveContainer" containerID="384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b"
Jul 21 18:51:01.892298 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:51:01.892279    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b\": container with ID starting with 384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b not found: ID does not exist" containerID="384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b"
Jul 21 18:51:01.892348 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.892308    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b"} err="failed to get container status \"384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b\": rpc error: code = NotFound desc = could not find container \"384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b\": container with ID starting with 384cefc36da94e9547028ef9d362410471df5c24a57d4e76e720d2f81212804b not found: ID does not exist"
Jul 21 18:51:01.892348 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.892326    2563 scope.go:117] "RemoveContainer" containerID="96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368"
Jul 21 18:51:01.892572 ip-10-0-129-22 kubenswrapper[2563]: E0721 18:51:01.892554    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368\": container with ID starting with 96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368 not found: ID does not exist" containerID="96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368"
Jul 21 18:51:01.892612 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:01.892578    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368"} err="failed to get container status \"96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368\": rpc error: code = NotFound desc = could not find container \"96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368\": container with ID starting with 96decf0081d364e6cf47ab5cb29607663ecf439cf52ffce2e898e288c3ff4368 not found: ID does not exist"
Jul 21 18:51:02.907688 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:02.907654    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" path="/var/lib/kubelet/pods/7673628b-2e58-4b30-ba38-5eb25b66dd83/volumes"
Jul 21 18:51:03.361219 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.361192    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff93ea2e-915b-4ca9-9605-8e3c9eff5c16/alertmanager/0.log"
Jul 21 18:51:03.400267 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.400245    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff93ea2e-915b-4ca9-9605-8e3c9eff5c16/config-reloader/0.log"
Jul 21 18:51:03.435275 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.435249    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff93ea2e-915b-4ca9-9605-8e3c9eff5c16/kube-rbac-proxy-web/0.log"
Jul 21 18:51:03.472189 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.472162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff93ea2e-915b-4ca9-9605-8e3c9eff5c16/kube-rbac-proxy/0.log"
Jul 21 18:51:03.508236 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.508211    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff93ea2e-915b-4ca9-9605-8e3c9eff5c16/kube-rbac-proxy-metric/0.log"
Jul 21 18:51:03.547958 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.547925    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff93ea2e-915b-4ca9-9605-8e3c9eff5c16/prom-label-proxy/0.log"
Jul 21 18:51:03.584297 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.584272    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_ff93ea2e-915b-4ca9-9605-8e3c9eff5c16/init-config-reloader/0.log"
Jul 21 18:51:03.641137 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.641061    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-4fj4r_b8e304a3-94a7-4671-a188-fbc216c91b6f/cluster-monitoring-operator/0.log"
Jul 21 18:51:03.670159 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.670135    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvxb8_40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae/kube-state-metrics/0.log"
Jul 21 18:51:03.699169 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.699142    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvxb8_40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae/kube-rbac-proxy-main/0.log"
Jul 21 18:51:03.725979 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.725955    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-cvxb8_40bc3b96-f384-4cac-89e9-ed1d3fb3f9ae/kube-rbac-proxy-self/0.log"
Jul 21 18:51:03.760827 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.760782    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7bc5f9ff5d-8fnrd_791f4d17-aa77-403b-a5a5-0314c63a72c7/metrics-server/0.log"
Jul 21 18:51:03.835009 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.834982    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4z6f4_51b3d631-7caa-471a-a098-4fa381ed02d7/node-exporter/0.log"
Jul 21 18:51:03.864382 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.864357    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4z6f4_51b3d631-7caa-471a-a098-4fa381ed02d7/kube-rbac-proxy/0.log"
Jul 21 18:51:03.893042 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:03.892962    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-4z6f4_51b3d631-7caa-471a-a098-4fa381ed02d7/init-textfile/0.log"
Jul 21 18:51:04.096244 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.096210    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-l4rs5_dad95311-bc9c-41dc-88b7-637efe0ff6f9/kube-rbac-proxy-main/0.log"
Jul 21 18:51:04.126356 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.126329    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-l4rs5_dad95311-bc9c-41dc-88b7-637efe0ff6f9/kube-rbac-proxy-self/0.log"
Jul 21 18:51:04.156140 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.156048    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-79f6958cdf-l4rs5_dad95311-bc9c-41dc-88b7-637efe0ff6f9/openshift-state-metrics/0.log"
Jul 21 18:51:04.208030 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.208003    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545df95-7386-4091-b684-ac65fce2638e/prometheus/0.log"
Jul 21 18:51:04.229455 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.229431    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545df95-7386-4091-b684-ac65fce2638e/config-reloader/0.log"
Jul 21 18:51:04.257917 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.257894    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545df95-7386-4091-b684-ac65fce2638e/thanos-sidecar/0.log"
Jul 21 18:51:04.294931 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.294907    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545df95-7386-4091-b684-ac65fce2638e/kube-rbac-proxy-web/0.log"
Jul 21 18:51:04.322887 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.322835    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545df95-7386-4091-b684-ac65fce2638e/kube-rbac-proxy/0.log"
Jul 21 18:51:04.352036 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.352007    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545df95-7386-4091-b684-ac65fce2638e/kube-rbac-proxy-thanos/0.log"
Jul 21 18:51:04.377979 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.377958    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_6545df95-7386-4091-b684-ac65fce2638e/init-config-reloader/0.log"
Jul 21 18:51:04.410567 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.410500    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bhsbb_cdf7bfbf-2443-411f-af32-4c6f2ee61b65/prometheus-operator/0.log"
Jul 21 18:51:04.433432 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.433408    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-bhsbb_cdf7bfbf-2443-411f-af32-4c6f2ee61b65/kube-rbac-proxy/0.log"
Jul 21 18:51:04.474622 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.474599    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-8wlkh_62bde93c-82cd-4ab9-8a93-b22780216f0f/prometheus-operator-admission-webhook/0.log"
Jul 21 18:51:04.515102 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.515078    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7759f6578c-bnrrl_7b85463b-9b67-435a-96d7-d7f0f52f0782/telemeter-client/0.log"
Jul 21 18:51:04.549849 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.549831    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7759f6578c-bnrrl_7b85463b-9b67-435a-96d7-d7f0f52f0782/reload/0.log"
Jul 21 18:51:04.574037 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:04.574020    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7759f6578c-bnrrl_7b85463b-9b67-435a-96d7-d7f0f52f0782/kube-rbac-proxy/0.log"
Jul 21 18:51:06.077545 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:06.077518    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-dp25d_699ff256-14c0-4378-8721-9a322e2dc0ec/networking-console-plugin/0.log"
Jul 21 18:51:07.872683 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.872654    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"]
Jul 21 18:51:07.873091 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.873000    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerName="gather"
Jul 21 18:51:07.873091 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.873013    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerName="gather"
Jul 21 18:51:07.873091 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.873021    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerName="copy"
Jul 21 18:51:07.873091 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.873026    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerName="copy"
Jul 21 18:51:07.873091 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.873079    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerName="gather"
Jul 21 18:51:07.873091 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.873090    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="7673628b-2e58-4b30-ba38-5eb25b66dd83" containerName="copy"
Jul 21 18:51:07.876087 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.876068    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:07.878843 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.878819    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4764r\"/\"openshift-service-ca.crt\""
Jul 21 18:51:07.880056 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.880031    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-4764r\"/\"kube-root-ca.crt\""
Jul 21 18:51:07.880153 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.880036    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-4764r\"/\"default-dockercfg-7dvvm\""
Jul 21 18:51:07.886183 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.885921    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"]
Jul 21 18:51:07.965539 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.965500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5sqr\" (UniqueName: \"kubernetes.io/projected/d47b9d86-41d6-4531-9ccc-b79006d8d321-kube-api-access-k5sqr\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:07.965539 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.965545    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-lib-modules\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:07.965739 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.965628    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-sys\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:07.965739 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.965653    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-proc\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:07.965739 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:07.965676    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-podres\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.066850 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.066818    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-sys\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.066978 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.066861    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-proc\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.066978 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.066877    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-podres\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.066978 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.066962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-proc\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.066978 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.066968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-sys\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.067277 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.066985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-podres\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.067277 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.067009    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5sqr\" (UniqueName: \"kubernetes.io/projected/d47b9d86-41d6-4531-9ccc-b79006d8d321-kube-api-access-k5sqr\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.067277 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.067038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-lib-modules\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.067277 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.067154    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d47b9d86-41d6-4531-9ccc-b79006d8d321-lib-modules\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.075322 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.075297    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5sqr\" (UniqueName: \"kubernetes.io/projected/d47b9d86-41d6-4531-9ccc-b79006d8d321-kube-api-access-k5sqr\") pod \"perf-node-gather-daemonset-p9879\" (UID: \"d47b9d86-41d6-4531-9ccc-b79006d8d321\") " pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.192356 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.192275    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.320550 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.320525    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"]
Jul 21 18:51:08.322754 ip-10-0-129-22 kubenswrapper[2563]: W0721 18:51:08.322722    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd47b9d86_41d6_4531_9ccc_b79006d8d321.slice/crio-90a657948b5add02b9e5e807a28efea4f86d01289ec31de09f640ccc61a95ea0 WatchSource:0}: Error finding container 90a657948b5add02b9e5e807a28efea4f86d01289ec31de09f640ccc61a95ea0: Status 404 returned error can't find the container with id 90a657948b5add02b9e5e807a28efea4f86d01289ec31de09f640ccc61a95ea0
Jul 21 18:51:08.689077 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.689038    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmpzt_e1a92e5b-3446-457c-bda1-e069f7fedc95/dns/0.log"
Jul 21 18:51:08.711873 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.711847    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qmpzt_e1a92e5b-3446-457c-bda1-e069f7fedc95/kube-rbac-proxy/0.log"
Jul 21 18:51:08.735652 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.735634    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-5njds_27821ce6-26d8-4a08-b587-91eeac050017/dns-node-resolver/0.log"
Jul 21 18:51:08.902163 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.902132    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879" event={"ID":"d47b9d86-41d6-4531-9ccc-b79006d8d321","Type":"ContainerStarted","Data":"43686cd7a528aee8ecae3aafb635ca153d3c3407fa444dfff4ce19e8a9799ccd"}
Jul 21 18:51:08.902593 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.902185    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879" event={"ID":"d47b9d86-41d6-4531-9ccc-b79006d8d321","Type":"ContainerStarted","Data":"90a657948b5add02b9e5e807a28efea4f86d01289ec31de09f640ccc61a95ea0"}
Jul 21 18:51:08.902593 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.902298    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:08.918999 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:08.918955    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879" podStartSLOduration=1.9189398519999998 podStartE2EDuration="1.918939852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-21 18:51:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-21 18:51:08.91812194 +0000 UTC m=+5128.476129046" watchObservedRunningTime="2026-07-21 18:51:08.918939852 +0000 UTC m=+5128.476946958"
Jul 21 18:51:09.366010 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:09.365981    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-p7867_ba892592-38ce-46ab-8073-907a0e715c39/node-ca/0.log"
Jul 21 18:51:10.206270 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:10.206225    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-7cfbdb577d-z6pf4_15e82759-7714-4b28-b4cb-c9289ae76056/router/0.log"
Jul 21 18:51:10.562521 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:10.562493    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-hqmpp_83f3e42b-06f3-4ffa-8270-a5b076a775fc/serve-healthcheck-canary/0.log"
Jul 21 18:51:11.157987 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:11.157914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl229_1350fcd2-f330-46c1-a54d-06f482e0800a/kube-rbac-proxy/0.log"
Jul 21 18:51:11.180622 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:11.180595    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl229_1350fcd2-f330-46c1-a54d-06f482e0800a/exporter/0.log"
Jul 21 18:51:11.204631 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:11.204614    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-dl229_1350fcd2-f330-46c1-a54d-06f482e0800a/extractor/0.log"
Jul 21 18:51:14.033454 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:14.033428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rrnnb_ee5c1828-9768-4bd9-833f-548609d7b59e/s3-init/0.log"
Jul 21 18:51:14.050220 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:14.050190    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-rrnnb_ee5c1828-9768-4bd9-833f-548609d7b59e/download-hf-model/0.log"
Jul 21 18:51:14.078354 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:14.078329    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-s7bk8_6cbaa73b-d112-4a01-8ef7-4217d26cebc1/s3-tls-init-custom/0.log"
Jul 21 18:51:14.095960 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:14.095938    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-s7bk8_6cbaa73b-d112-4a01-8ef7-4217d26cebc1/download-hf-model/0.log"
Jul 21 18:51:14.129936 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:14.129912    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-92qlj_4d270610-b9ca-435c-80c6-a8e3456c0886/s3-tls-init-serving/0.log"
Jul 21 18:51:14.148553 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:14.148525    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-92qlj_4d270610-b9ca-435c-80c6-a8e3456c0886/download-hf-model/0.log"
Jul 21 18:51:14.916207 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:14.916182    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-4764r/perf-node-gather-daemonset-p9879"
Jul 21 18:51:19.772341 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:19.772316    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xzbr_77eb9156-b7f3-4f07-88f2-c5a2329e7bc4/kube-multus-additional-cni-plugins/0.log"
Jul 21 18:51:19.797489 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:19.797462    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xzbr_77eb9156-b7f3-4f07-88f2-c5a2329e7bc4/egress-router-binary-copy/0.log"
Jul 21 18:51:19.819422 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:19.819397    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xzbr_77eb9156-b7f3-4f07-88f2-c5a2329e7bc4/cni-plugins/0.log"
Jul 21 18:51:19.842623 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:19.842556    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xzbr_77eb9156-b7f3-4f07-88f2-c5a2329e7bc4/bond-cni-plugin/0.log"
Jul 21 18:51:19.863754 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:19.863733    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xzbr_77eb9156-b7f3-4f07-88f2-c5a2329e7bc4/routeoverride-cni/0.log"
Jul 21 18:51:19.885707 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:19.885687    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xzbr_77eb9156-b7f3-4f07-88f2-c5a2329e7bc4/whereabouts-cni-bincopy/0.log"
Jul 21 18:51:19.910026 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:19.910009    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-5xzbr_77eb9156-b7f3-4f07-88f2-c5a2329e7bc4/whereabouts-cni/0.log"
Jul 21 18:51:20.264872 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:20.264845    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-lgh5k_fe6ad5b4-7a10-49b2-9618-440018cb51c3/kube-multus/0.log"
Jul 21 18:51:20.460943 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:20.460919    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rz8hm_8cfe68ed-2ecc-4369-adcd-235f6b199b24/network-metrics-daemon/0.log"
Jul 21 18:51:20.482701 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:20.482677    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-rz8hm_8cfe68ed-2ecc-4369-adcd-235f6b199b24/kube-rbac-proxy/0.log"
Jul 21 18:51:21.729415 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.729347    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-controller/0.log"
Jul 21 18:51:21.746704 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.746683    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/1.log"
Jul 21 18:51:21.746845 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.746742    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovn-acl-logging/0.log"
Jul 21 18:51:21.765680 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.765663    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/kube-rbac-proxy-node/0.log"
Jul 21 18:51:21.788815 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.788781    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/kube-rbac-proxy-ovn-metrics/0.log"
Jul 21 18:51:21.811184 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.811162    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/northd/0.log"
Jul 21 18:51:21.837338 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.837321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/nbdb/0.log"
Jul 21 18:51:21.859345 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.859330    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/sbdb/0.log"
Jul 21 18:51:21.907839 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:21.907818    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-shlvf_3a3b0072-0f03-49d4-b4cc-b69dab9d6f1e/ovnkube-controller/0.log"
Jul 21 18:51:22.952857 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:22.952829    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-7bc5b6bf85-l5q5z_87803527-5457-4f07-9fa4-54080f6959a2/check-endpoints/0.log"
Jul 21 18:51:23.001992 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:23.001965    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-qq9ln_ff525c04-45a3-46cd-883e-fbbc3416f73c/network-check-target-container/0.log"
Jul 21 18:51:24.009349 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:24.009302    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-5wmkn_b64a4088-eec0-4e59-bb26-7b4297a68fdf/iptables-alerter/0.log"
Jul 21 18:51:24.762750 ip-10-0-129-22 kubenswrapper[2563]: I0721 18:51:24.762722    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-tl99f_a4b01391-be0e-4c48-a84d-2842bedb9248/tuned/0.log"