Jul 31 14:46:24.419707 ip-10-0-129-192 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 31 14:46:24.419719 ip-10-0-129-192 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 31 14:46:24.419729 ip-10-0-129-192 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 31 14:46:24.420090 ip-10-0-129-192 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 31 14:46:34.447619 ip-10-0-129-192 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 31 14:46:34.447642 ip-10-0-129-192 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 825a0500f0364f08ace54c32e95e658c --
Jul 31 14:48:22.322033 ip-10-0-129-192 systemd[1]: Starting Kubernetes Kubelet...
Jul 31 14:48:23.771256 ip-10-0-129-192 kubenswrapper[2561]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:23.771256 ip-10-0-129-192 kubenswrapper[2561]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 31 14:48:23.771256 ip-10-0-129-192 kubenswrapper[2561]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:23.771256 ip-10-0-129-192 kubenswrapper[2561]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 31 14:48:23.771256 ip-10-0-129-192 kubenswrapper[2561]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 31 14:48:23.771946 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.771322    2561 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778805    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778832    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778836    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778840    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778843    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778846    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778849    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778852    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.778838 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778855    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778858    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778861    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778864    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778866    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778869    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778871    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778874    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778877    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778879    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778882    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778884    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778888    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778891    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778893    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778896    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778899    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778908    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778911    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778914    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.779250 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778917    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778919    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778930    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778933    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778937    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778939    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778942    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778947    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778951    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778955    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778958    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778961    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778964    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778967    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778970    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778974    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778977    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778980    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778982    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.779728 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778986    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778989    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778992    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778996    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.778999    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779002    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779005    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779008    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779011    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779013    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779016    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779019    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779022    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779025    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779027    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779033    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779037    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779040    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779043    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.780230 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779046    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779049    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779052    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779055    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779058    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779061    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779063    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779066    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779069    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779072    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779075    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779077    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779080    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779082    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779085    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779088    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779090    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779094    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779096    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779099    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.780686 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779101    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779104    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779107    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779109    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779112    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779114    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779117    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779119    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779122    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779124    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779127    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779131    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779134    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779910    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779918    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779921    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779924    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779927    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779929    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.781194 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779932    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779936    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779939    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779941    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779944    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779947    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779949    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779952    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779955    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779957    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779960    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779964    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779967    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779971    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779976    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779979    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779982    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779985    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779989    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779992    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.781653 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779995    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.779998    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780000    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780003    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780006    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780008    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780011    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780014    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780016    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780019    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780022    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780024    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780027    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780031    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780033    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780036    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780038    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780041    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780044    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780047    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.782146 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780049    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780052    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780054    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780057    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780060    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780062    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780066    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780069    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780072    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780075    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780077    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780080    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780082    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780085    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780087    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780090    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780092    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780098    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780101    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.782659 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780103    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780106    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780108    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780111    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780114    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780117    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780120    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780123    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780126    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780128    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780131    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780134    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780136    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780139    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780141    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780144    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780146    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780150    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780168    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780171    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.783142 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780174    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780177    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780180    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780183    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780186    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780188    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780191    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780195    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780198    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780201    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780206    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780209    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780212    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780215    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780691    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780697    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780700    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780704    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780707    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.783664 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780710    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780713    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780716    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780719    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780722    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780724    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780727    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780730    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780733    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780735    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780738    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780741    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780744    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780747    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780749    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780752    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780754    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780757    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780760    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.784132 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780763    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780765    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780768    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780770    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780779    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780781    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780784    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780787    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780790    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780793    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780795    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780798    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780801    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780804    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780806    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780809    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780811    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780815    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780819    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780823    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.784649 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780826    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780829    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780832    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780835    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780838    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780841    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780844    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780847    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780849    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780852    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780854    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780858    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780871    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780875    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780879    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780882    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780886    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780889    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780892    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.785162 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780894    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780897    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780899    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780902    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780904    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780907    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780910    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780912    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780915    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780918    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780920    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780923    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780928    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780930    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780933    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780936    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780938    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780940    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780943    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780946    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.785660 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780949    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780951    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780954    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780957    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780959    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780962    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780964    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780968    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780971    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780975    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780978    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780980    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780983    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780986    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780988    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.780991    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781068    2561 flags.go:64] FLAG: --address="0.0.0.0"
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781078    2561 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781084    2561 flags.go:64] FLAG: --anonymous-auth="true"
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781089    2561 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781093    2561 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 31 14:48:23.786151 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781097    2561 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781101    2561 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781105    2561 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781109    2561 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781115    2561 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781120    2561 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781123    2561 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781127    2561 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781130    2561 flags.go:64] FLAG: --cgroup-root=""
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781134    2561 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781137    2561 flags.go:64] FLAG: --client-ca-file=""
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781140    2561 flags.go:64] FLAG: --cloud-provider="external"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781143    2561 flags.go:64] FLAG: --cluster-dns="[]"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781148    2561 flags.go:64] FLAG: --cluster-domain=""
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781164    2561 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781167    2561 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781170    2561 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781174    2561 flags.go:64] FLAG: --container-log-max-files="5"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781178    2561 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781181    2561 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781186    2561 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781189    2561 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781193    2561 flags.go:64] FLAG: --contention-profiling="false"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781196    2561 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 31 14:48:23.786717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781199    2561 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781203    2561 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781206    2561 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781210    2561 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781214    2561 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781218    2561 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781221    2561 flags.go:64] FLAG: --enable-load-reader="false"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781224    2561 flags.go:64] FLAG: --enable-server="true"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781227    2561 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781231    2561 flags.go:64] FLAG: --event-burst="100"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781234    2561 flags.go:64] FLAG: --event-qps="50"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781237    2561 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781240    2561 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781243    2561 flags.go:64] FLAG: --eviction-hard=""
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781250    2561 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781254    2561 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781257    2561 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781260    2561 flags.go:64] FLAG: --eviction-soft=""
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781263    2561 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781266    2561 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781270    2561 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781273    2561 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781276    2561 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781279    2561 flags.go:64] FLAG: --fail-swap-on="true"
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781282    2561 flags.go:64] FLAG: --feature-gates=""
Jul 31 14:48:23.787328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781286    2561 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781289    2561 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781292    2561 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781296    2561 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781299    2561 flags.go:64] FLAG: --healthz-port="10248"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781302    2561 flags.go:64] FLAG: --help="false"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781305    2561 flags.go:64] FLAG: --hostname-override="ip-10-0-129-192.ec2.internal"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781308    2561 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781311    2561 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781314    2561 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781317    2561 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781324    2561 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781327    2561 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781330    2561 flags.go:64] FLAG: --image-service-endpoint=""
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781333    2561 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781336    2561 flags.go:64] FLAG: --kube-api-burst="100"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781339    2561 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781342    2561 flags.go:64] FLAG: --kube-api-qps="50"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781345    2561 flags.go:64] FLAG: --kube-reserved=""
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781348    2561 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781351    2561 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781354    2561 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781357    2561 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781370    2561 flags.go:64] FLAG: --lock-file=""
Jul 31 14:48:23.787994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781372    2561 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781376    2561 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781379    2561 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781385    2561 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781388    2561 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781391    2561 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781394    2561 flags.go:64] FLAG: --logging-format="text"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781397    2561 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781401    2561 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781404    2561 flags.go:64] FLAG: --manifest-url=""
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781407    2561 flags.go:64] FLAG: --manifest-url-header=""
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781411    2561 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781414    2561 flags.go:64] FLAG: --max-open-files="1000000"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781419    2561 flags.go:64] FLAG: --max-pods="110"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781422    2561 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781426    2561 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781429    2561 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781432    2561 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781435    2561 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781440    2561 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781443    2561 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781451    2561 flags.go:64] FLAG: --node-status-max-images="50"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781454    2561 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781458    2561 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 31 14:48:23.789116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781461    2561 flags.go:64] FLAG: --pod-cidr=""
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781464    2561 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:267c8c33005f5dc3d4121af550b1c396e30e2928a13624d7362adfbfb7f440f8"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781469    2561 flags.go:64] FLAG: --pod-manifest-path=""
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781472    2561 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781476    2561 flags.go:64] FLAG: --pods-per-core="0"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781479    2561 flags.go:64] FLAG: --port="10250"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781482    2561 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781485    2561 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a5ba61d924b3b42c"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781488    2561 flags.go:64] FLAG: --qos-reserved=""
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781491    2561 flags.go:64] FLAG: --read-only-port="10255"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781494    2561 flags.go:64] FLAG: --register-node="true"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781497    2561 flags.go:64] FLAG: --register-with-taints=""
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781501    2561 flags.go:64] FLAG: --registry-burst="10"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781505    2561 flags.go:64] FLAG: --registry-qps="5"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781508    2561 flags.go:64] FLAG: --reserved-cpus=""
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781511    2561 flags.go:64] FLAG: --reserved-memory=""
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781515    2561 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781518    2561 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781521    2561 flags.go:64] FLAG: --rotate-certificates="false"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781524    2561 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781527    2561 flags.go:64] FLAG: --runonce="false"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781530    2561 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781533    2561 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781536    2561 flags.go:64] FLAG: --seccomp-default="false"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781539    2561 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 31 14:48:23.790228 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781542    2561 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781545    2561 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781548    2561 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781553    2561 flags.go:64] FLAG: --storage-driver-password="root"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781556    2561 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781559    2561 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781562    2561 flags.go:64] FLAG: --storage-driver-user="root"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781564    2561 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781567    2561 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781571    2561 flags.go:64] FLAG: --system-cgroups=""
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781573    2561 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781578    2561 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781581    2561 flags.go:64] FLAG: --tls-cert-file=""
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781584    2561 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781588    2561 flags.go:64] FLAG: --tls-min-version=""
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781591    2561 flags.go:64] FLAG: --tls-private-key-file=""
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781594    2561 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781597    2561 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781600    2561 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781604    2561 flags.go:64] FLAG: --v="2"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781608    2561 flags.go:64] FLAG: --version="false"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781614    2561 flags.go:64] FLAG: --vmodule=""
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781617    2561 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.781621    2561 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781740    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.791432 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781744    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781747    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781749    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781752    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781754    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781757    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781760    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781762    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781765    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781767    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781771    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781774    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781776    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781779    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781781    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781784    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781786    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781789    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781792    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781794    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.792368 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781797    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781799    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781802    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781805    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781807    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781810    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781814    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781818    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781822    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781827    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781830    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781833    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781836    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781838    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781841    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781843    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781846    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781848    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781851    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.793030 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781853    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781856    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781858    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781862    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781865    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781867    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781870    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781872    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781875    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781878    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781882    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781885    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781888    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781890    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781893    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781895    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781898    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781900    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781903    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781907    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.793842 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781910    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781912    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781915    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781918    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781921    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781923    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781926    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781928    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781931    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781933    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781936    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781938    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781941    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781944    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781946    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781950    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781953    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781956    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781958    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781961    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.794546 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781964    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781966    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781969    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781971    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781973    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781976    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781979    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781981    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781984    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781986    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781989    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781994    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.781997    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.782000    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.782002    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.782006    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.782010    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.782013    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.795208 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.782016    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.782723    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.790480    2561 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.790506    2561 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.790535    2561 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.790549    2561 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790643    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790651    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790656    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790662    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790666    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790672    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790701    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790709    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790714    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.795685 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790719    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790724    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790729    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790734    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790739    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790743    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790748    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790754    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790761    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790766    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790771    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790775    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790779    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790784    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790788    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790794    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790798    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790803    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790806    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790812    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.796056 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790817    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790821    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790825    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790830    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790834    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790839    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790843    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790850    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790856    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790861    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790866    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790872    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790876    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790882    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790887    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790892    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790896    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790900    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790905    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.796638 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790909    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790914    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790918    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790922    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790926    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790930    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790934    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790938    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790941    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790947    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790951    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790955    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790960    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790965    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790969    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790973    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790978    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790982    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790986    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790990    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.797175 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790994    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.790998    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791003    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791007    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791012    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791015    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791019    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791024    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791028    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791032    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791038    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791043    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791047    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791052    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791058    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791062    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791066    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791071    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791075    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.797732 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791080    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791084    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791088    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791093    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791098    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791103    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791108    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791113    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791117    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791122    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791126    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791130    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.791139    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791363    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791371    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 31 14:48:23.798223 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791376    2561 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791380    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791386    2561 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791391    2561 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791395    2561 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791400    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791404    2561 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791408    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791412    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791417    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791421    2561 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791425    2561 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791429    2561 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791433    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791437    2561 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791441    2561 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791445    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791449    2561 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791453    2561 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791456    2561 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 31 14:48:23.798635 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791460    2561 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791465    2561 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791470    2561 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791475    2561 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791480    2561 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791485    2561 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791489    2561 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791494    2561 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791498    2561 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791503    2561 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791507    2561 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791511    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791515    2561 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791519    2561 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791523    2561 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791527    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791532    2561 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791536    2561 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791540    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791545    2561 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 31 14:48:23.799321 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791549    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791553    2561 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791557    2561 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791561    2561 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791565    2561 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791569    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791573    2561 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791578    2561 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791582    2561 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791586    2561 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791590    2561 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791595    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791599    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791603    2561 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791609    2561 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791614    2561 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791621    2561 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791627    2561 feature_gate.go:328] unrecognized feature gate: Example2
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791632    2561 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 31 14:48:23.799988 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791637    2561 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791643    2561 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791648    2561 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791653    2561 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791660    2561 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791665    2561 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791670    2561 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791675    2561 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791680    2561 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791684    2561 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791689    2561 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791693    2561 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791697    2561 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791701    2561 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791706    2561 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791710    2561 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791714    2561 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791719    2561 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791723    2561 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 31 14:48:23.800538 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791727    2561 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791731    2561 feature_gate.go:328] unrecognized feature gate: Example
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791735    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791739    2561 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791743    2561 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791747    2561 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791752    2561 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791757    2561 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791761    2561 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791767    2561 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791771    2561 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791776    2561 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791780    2561 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791784    2561 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791788    2561 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791792    2561 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791796    2561 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791800    2561 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 31 14:48:23.801035 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:23.791804    2561 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 31 14:48:23.801515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.791813    2561 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 31 14:48:23.801515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.792010    2561 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 31 14:48:23.801515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.794818    2561 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 31 14:48:23.801515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.796647    2561 server.go:1019] "Starting client certificate rotation"
Jul 31 14:48:23.801515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.796753    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:48:23.801515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.796794    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 31 14:48:23.801515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.799530    2561 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:48:23.806300 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.806231    2561 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 31 14:48:23.820312 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.820285    2561 log.go:25] "Validated CRI v1 runtime API"
Jul 31 14:48:23.822933 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.822904    2561 log.go:25] "Validated CRI v1 image API"
Jul 31 14:48:23.823741 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.823723    2561 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 31 14:48:23.826425 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.826404    2561 fs.go:135] Filesystem UUIDs: map[27b2b92c-41a4-41dd-9d0f-d27208c3ca67:/dev/nvme0n1p3 66be83cc-50b6-4a49-a570-f94a457e3605:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Jul 31 14:48:23.826490 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.826426    2561 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 31 14:48:23.827799 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.827781    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:23.832516 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.832388    2561 manager.go:216] Machine: {Timestamp:2026-07-31 14:48:23.831095817 +0000 UTC m=+0.350519847 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101542 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f7e682fc3d3b4381136bfc37c8bbc SystemUUID:ec2f7e68-2fc3-d3b4-3811-36bfc37c8bbc BootID:825a0500-f036-4f08-ace5-4c32e95e658c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:06:c1:31:ba:11 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:06:c1:31:ba:11 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:82:ca:83:1b:fe:e7 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 31 14:48:23.832516 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.832503    2561 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 31 14:48:23.832672 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.832600    2561 manager.go:232] Version: {KernelVersion:5.14.0-570.131.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260727-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 31 14:48:23.835049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.835015    2561 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 31 14:48:23.835262 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.835052    2561 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-192.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 31 14:48:23.835915 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.835901    2561 topology_manager.go:138] "Creating topology manager with none policy"
Jul 31 14:48:23.835979 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.835922    2561 container_manager_linux.go:309] "Creating device plugin manager"
Jul 31 14:48:23.835979 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.835933    2561 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:48:23.837649 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.837633    2561 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 31 14:48:23.838445 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.838433    2561 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 31 14:48:23.842030 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842010    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:48:23.842197 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842185    2561 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 31 14:48:23.842269 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842239    2561 kubelet.go:486] "Attempting to sync node with API server"
Jul 31 14:48:23.842269 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842255    2561 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 31 14:48:23.842363 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842272    2561 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 31 14:48:23.842363 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842286    2561 kubelet.go:398] "Adding apiserver pod source"
Jul 31 14:48:23.842363 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842302    2561 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 31 14:48:23.842780 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842766    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:48:23.842851 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.842789    2561 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 31 14:48:23.845143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.845119    2561 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 31 14:48:23.846571 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.846557    2561 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 31 14:48:23.846629 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.846588    2561 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 31 14:48:23.847649 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847636    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 31 14:48:23.847690 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847652    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 31 14:48:23.847690 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847658    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 31 14:48:23.847690 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847664    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 31 14:48:23.847690 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847673    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 31 14:48:23.847690 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847679    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 31 14:48:23.847690 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847687    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 31 14:48:23.847690 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847692    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 31 14:48:23.847870 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847700    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 31 14:48:23.847870 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847705    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 31 14:48:23.847870 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847719    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 31 14:48:23.847870 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.847724    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 31 14:48:23.848517 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.848504    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 31 14:48:23.848517 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.848515    2561 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 31 14:48:23.851889 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.851872    2561 server.go:1268] "Started kubelet"
Jul 31 14:48:23.851988 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.851960    2561 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 31 14:48:23.852035 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.851977    2561 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 31 14:48:23.852035 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.852015    2561 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 31 14:48:23.852832 ip-10-0-129-192 systemd[1]: Started Kubernetes Kubelet.
Jul 31 14:48:23.856556 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.856145    2561 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 31 14:48:23.856556 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.856227    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7b84s"
Jul 31 14:48:23.858680 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.858650    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 31 14:48:23.858680 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.858634    2561 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-192.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 31 14:48:23.858826 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.858685    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 31 14:48:23.858875 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.858862    2561 server.go:310] "Adding debug handlers to kubelet server"
Jul 31 14:48:23.859020 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.858991    2561 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 31 14:48:23.861565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.861541    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-7b84s"
Jul 31 14:48:23.862817 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.862800    2561 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 31 14:48:23.862890 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.862847    2561 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:23.862977 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.862946    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:23.863382 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.863341    2561 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 31 14:48:23.863485 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.863436    2561 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 31 14:48:23.863485 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.863451    2561 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 31 14:48:23.863664 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.863649    2561 reconstruct.go:97] "Volume reconstruction finished"
Jul 31 14:48:23.863730 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.863666    2561 reconciler.go:29] "Reconciler: start to sync state"
Jul 31 14:48:23.863800 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.863781    2561 factory.go:55] Registering systemd factory
Jul 31 14:48:23.863800 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.863801    2561 factory.go:223] Registration of the systemd container factory successfully
Jul 31 14:48:23.864070 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.864054    2561 factory.go:153] Registering CRI-O factory
Jul 31 14:48:23.864134 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.864075    2561 factory.go:223] Registration of the crio container factory successfully
Jul 31 14:48:23.864134 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.864124    2561 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 31 14:48:23.864251 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.864147    2561 factory.go:103] Registering Raw factory
Jul 31 14:48:23.864251 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.864179    2561 manager.go:1196] Started watching for new ooms in manager
Jul 31 14:48:23.865288 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.865264    2561 manager.go:349] Starting recovery of all containers
Jul 31 14:48:23.867274 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.867250    2561 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 31 14:48:23.867369 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.867313    2561 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-192.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 31 14:48:23.868078 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.867166    2561 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-192.ec2.internal.18c76740773e8618  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-192.ec2.internal,UID:ip-10-0-129-192.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-192.ec2.internal,},FirstTimestamp:2026-07-31 14:48:23.851845144 +0000 UTC m=+0.371269174,LastTimestamp:2026-07-31 14:48:23.851845144 +0000 UTC m=+0.371269174,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-192.ec2.internal,}"
Jul 31 14:48:23.876791 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.876774    2561 manager.go:354] Recovery completed
Jul 31 14:48:23.881594 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.881455    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:23.883924 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.883901    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:23.884013 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.883940    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:23.884013 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.883951    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:23.884472 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.884456    2561 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 31 14:48:23.884472 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.884466    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:23.884472 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.884469    2561 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 31 14:48:23.884620 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.884490    2561 state_mem.go:36] "Initialized new in-memory state store"
Jul 31 14:48:23.885299 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.885282    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:23.885375 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.885314    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:23.885375 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.885327    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:23.885607 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.885594    2561 policy_none.go:49] "None policy: Start"
Jul 31 14:48:23.885665 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.885613    2561 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 31 14:48:23.885665 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.885648    2561 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 31 14:48:23.886626 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.886614    2561 policy_none.go:47] "Start"
Jul 31 14:48:23.929841 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.929781    2561 manager.go:337] "Starting Device Plugin manager"
Jul 31 14:48:23.929841 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.929824    2561 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 31 14:48:23.929841 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.929835    2561 server.go:87] "Starting device plugin registration server"
Jul 31 14:48:23.930178 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.930147    2561 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 31 14:48:23.930217 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.930185    2561 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 31 14:48:23.930291 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.930272    2561 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 31 14:48:23.930418 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.930374    2561 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 31 14:48:23.930418 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.930389    2561 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 31 14:48:23.930939 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.930922    2561 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 31 14:48:23.931005 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.930966    2561 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:23.954205 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.954174    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 31 14:48:23.955798 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.955310    2561 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 31 14:48:23.955798 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.955347    2561 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 31 14:48:23.955798 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.955373    2561 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 31 14:48:23.955798 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:23.955408    2561 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 31 14:48:23.957882 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:23.957861    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:24.031251 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.031203    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.032292 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.032274    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.032375 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.032311    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.032375 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.032322    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.032375 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.032348    2561 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.041202 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.041183    2561 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.041256 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.041210    2561 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-192.ec2.internal\": node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.055775 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.055746    2561 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"]
Jul 31 14:48:24.055884 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.055815    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.058661 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.058643    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.058764 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.058677    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.058764 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.058691    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.060084 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.060067    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.060210 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.060195    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.060267 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.060228    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.060442 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.060429    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.061226 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.061210    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.061339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.061238    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.061339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.061256    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.061339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.061293    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.061339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.061314    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.061339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.061328    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.062497 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.062480    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.062586 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.062507    2561 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 31 14:48:24.063869 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.063847    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientMemory"
Jul 31 14:48:24.063948 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.063873    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasNoDiskPressure"
Jul 31 14:48:24.063948 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.063886    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeHasSufficientPID"
Jul 31 14:48:24.064295 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.064275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/085be5e42d0150ff2c7d62ba258a4275-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal\" (UID: \"085be5e42d0150ff2c7d62ba258a4275\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.064370 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.064307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085be5e42d0150ff2c7d62ba258a4275-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal\" (UID: \"085be5e42d0150ff2c7d62ba258a4275\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.064370 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.064334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645b936c36ac9f10f261c827f21930d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.ec2.internal\" (UID: \"645b936c36ac9f10f261c827f21930d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.079781 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.079750    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-192.ec2.internal\" not found" node="ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.083820 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.083801    2561 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-192.ec2.internal\" not found" node="ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.160515 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.160475    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.164813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.164793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645b936c36ac9f10f261c827f21930d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.ec2.internal\" (UID: \"645b936c36ac9f10f261c827f21930d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.164875 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.164821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/085be5e42d0150ff2c7d62ba258a4275-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal\" (UID: \"085be5e42d0150ff2c7d62ba258a4275\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.164875 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.164845    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085be5e42d0150ff2c7d62ba258a4275-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal\" (UID: \"085be5e42d0150ff2c7d62ba258a4275\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.164954 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.164894    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/085be5e42d0150ff2c7d62ba258a4275-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal\" (UID: \"085be5e42d0150ff2c7d62ba258a4275\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.164954 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.164913    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/085be5e42d0150ff2c7d62ba258a4275-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal\" (UID: \"085be5e42d0150ff2c7d62ba258a4275\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.164954 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.164915    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/645b936c36ac9f10f261c827f21930d7-config\") pod \"kube-apiserver-proxy-ip-10-0-129-192.ec2.internal\" (UID: \"645b936c36ac9f10f261c827f21930d7\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.260981 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.260915    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.361671 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.361626    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.382130 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.382100    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.386550 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.386422    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:24.462285 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.462230    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.562840 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.562774    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.663269 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.663220    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.763682 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.763651    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.796036 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.796008    2561 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 31 14:48:24.796452 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.796206    2561 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 31 14:48:24.815308 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.815248    2561 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:24.863803 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.863773    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.865985 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.865959    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-30 14:43:23 +0000 UTC" deadline="2027-12-28 03:23:59.391262151 +0000 UTC"
Jul 31 14:48:24.865985 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.865984    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12348h35m34.525280758s"
Jul 31 14:48:24.964765 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:24.964738    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:24.985532 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:24.985496    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod085be5e42d0150ff2c7d62ba258a4275.slice/crio-cc741f55fc84ec07f82da362b2398c443ab95fe8e65e509b9676202f12768f5d WatchSource:0}: Error finding container cc741f55fc84ec07f82da362b2398c443ab95fe8e65e509b9676202f12768f5d: Status 404 returned error can't find the container with id cc741f55fc84ec07f82da362b2398c443ab95fe8e65e509b9676202f12768f5d
Jul 31 14:48:24.985885 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:24.985858    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod645b936c36ac9f10f261c827f21930d7.slice/crio-6a7667fa77347035724983c890fb7759dd49da30cdad93524c5925ee1e4f909a WatchSource:0}: Error finding container 6a7667fa77347035724983c890fb7759dd49da30cdad93524c5925ee1e4f909a: Status 404 returned error can't find the container with id 6a7667fa77347035724983c890fb7759dd49da30cdad93524c5925ee1e4f909a
Jul 31 14:48:24.992892 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:24.992877    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:48:25.019274 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.019253    2561 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:25.065100 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:25.065077    2561 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-192.ec2.internal\" not found"
Jul 31 14:48:25.143098 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.143019    2561 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:25.163992 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.163966    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:25.174581 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.174556    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:48:25.177371 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.177357    2561 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal"
Jul 31 14:48:25.193108 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.193091    2561 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 31 14:48:25.844015 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.843984    2561 apiserver.go:52] "Watching apiserver"
Jul 31 14:48:25.853884 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.853860    2561 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 31 14:48:25.855738 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.855713    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-gfczg","openshift-cluster-node-tuning-operator/tuned-dgg8r","openshift-dns/node-resolver-wr8xz","openshift-image-registry/node-ca-lz89t","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal","openshift-multus/multus-additional-cni-plugins-7ctdq","openshift-multus/multus-cx6kt","openshift-network-operator/iptables-alerter-9gzdg","kube-system/konnectivity-agent-9j6k6","kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh","openshift-multus/network-metrics-daemon-qpj79","openshift-network-diagnostics/network-check-target-ch9qv","openshift-ovn-kubernetes/ovnkube-node-k2dtc"]
Jul 31 14:48:25.857304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.857279    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.858552 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.858524    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.859435 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.859415    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-9hzt8\""
Jul 31 14:48:25.859526 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.859442    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 31 14:48:25.859526 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.859417    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 31 14:48:25.859827 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.859790    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 31 14:48:25.859917 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.859868    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 31 14:48:25.859917 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.859892    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.860508 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.860488    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 31 14:48:25.860885 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.860847    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 31 14:48:25.860885 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.860849    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 31 14:48:25.861053 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.860847    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-rr9h5\""
Jul 31 14:48:25.861205 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.861185    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.861726 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.861708    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wfkfx\""
Jul 31 14:48:25.861816 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.861767    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 31 14:48:25.861976 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.861963    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 31 14:48:25.862677 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.862657    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.863292 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.863251    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 31 14:48:25.863292 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.863289    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 31 14:48:25.863444 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.863318    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 31 14:48:25.863444 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.863377    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-s4qdc\""
Jul 31 14:48:25.863830 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.863811    2561 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 31 14:48:25.864131 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.864114    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.865052 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.865033    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 31 14:48:25.865142 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.865065    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 31 14:48:25.865142 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.865119    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 31 14:48:25.865274 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.865163    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 31 14:48:25.865720 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.865691    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-wv8q6\""
Jul 31 14:48:25.865888 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.865874    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 31 14:48:25.866232 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.866206    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:25.866662 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.866646    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 31 14:48:25.866740 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.866667    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 31 14:48:25.866740 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.866647    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 31 14:48:25.866986 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.866965    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ctwcr\""
Jul 31 14:48:25.868100 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.868083    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:25.868204 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:25.868182    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:25.869333 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.869314    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lvzsk\""
Jul 31 14:48:25.869424 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.869325    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 31 14:48:25.869560 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.869540    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 31 14:48:25.873020 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-kubelet\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.873117 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-tmp-dir\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.873117 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sglvr\" (UniqueName: \"kubernetes.io/projected/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-kube-api-access-sglvr\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.873117 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873075    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-log-socket\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873285 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873127    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-cni-netd\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873285 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873146    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/598105ef-f9a1-43c1-abf2-827929b9291b-ovn-node-metrics-cert\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873285 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873183    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68e2f225-4cc2-465b-b01b-6f1e8142b94b-host\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.873285 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873223    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjthp\" (UniqueName: \"kubernetes.io/projected/68e2f225-4cc2-465b-b01b-6f1e8142b94b-kube-api-access-fjthp\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.873285 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873248    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-systemd-units\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873285 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873285    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-etc-selinux\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873307    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-device-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873325    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-run-netns\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873363    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-system-cni-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873397    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-cnibin\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873430    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-netns\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873456    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873481    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-ovn\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873504    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-node-log\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873543    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873577    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rdz4\" (UniqueName: \"kubernetes.io/projected/bf493cbb-ad21-4177-b194-d73d297a4cb0-kube-api-access-9rdz4\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-cni-multus\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873644    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-daemon-config\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873665    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-socket-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873688    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-systemd\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873737    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-ovnkube-config\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pfl8\" (UniqueName: \"kubernetes.io/projected/598105ef-f9a1-43c1-abf2-827929b9291b-kube-api-access-8pfl8\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873812    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-os-release\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873829    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-socket-dir-parent\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-cni-bin\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.873926 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68e2f225-4cc2-465b-b01b-6f1e8142b94b-serviceca\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.873977    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-cni-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874003    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-k8s-cni-cncf-io\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dws66\" (UniqueName: \"kubernetes.io/projected/2203e737-ecf1-46d7-81ea-be5ee9c0d476-kube-api-access-dws66\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qb252\" (UniqueName: \"kubernetes.io/projected/7b9c89ab-1a88-40b8-90d2-8c7230405e74-kube-api-access-qb252\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874077    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-env-overrides\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874109    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-cnibin\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874133    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2203e737-ecf1-46d7-81ea-be5ee9c0d476-cni-binary-copy\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-registration-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874211    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-os-release\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874232    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-hostroot\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874254    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-conf-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874279    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-var-lib-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874306    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-cni-bin\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-sys-fs\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.874419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874348    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-kubelet\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874362    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-slash\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874383    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874422    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874447    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-etc-kubernetes\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874479    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-hosts-file\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874494    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-ovnkube-script-lib\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-multus-certs\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874537    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-etc-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.874571    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-system-cni-dir\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.875111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.875039    2561 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 31 14:48:25.875590 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.875509    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.877250 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.877230    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:25.881389 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.879827    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:25.881389 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.880525    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:25.881389 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:25.880774    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:25.881389 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.881107    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 31 14:48:25.881389 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.881257    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7fcf7\""
Jul 31 14:48:25.881676 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.881449    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-pkb8m\""
Jul 31 14:48:25.881676 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.881486    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:25.881676 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.881552    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 31 14:48:25.881802 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.881776    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:48:25.883200 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.883179    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:25.883289 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:25.883265    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:25.907328 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.907305    2561 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-62psp"
Jul 31 14:48:25.915012 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.914991    2561 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-62psp"
Jul 31 14:48:25.960379 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.960330    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal" event={"ID":"085be5e42d0150ff2c7d62ba258a4275","Type":"ContainerStarted","Data":"cc741f55fc84ec07f82da362b2398c443ab95fe8e65e509b9676202f12768f5d"}
Jul 31 14:48:25.961467 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.961440    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal" event={"ID":"645b936c36ac9f10f261c827f21930d7","Type":"ContainerStarted","Data":"6a7667fa77347035724983c890fb7759dd49da30cdad93524c5925ee1e4f909a"}
Jul 31 14:48:25.964461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.964441    2561 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 31 14:48:25.974910 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.974882    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68e2f225-4cc2-465b-b01b-6f1e8142b94b-host\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.975071 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.974965    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/68e2f225-4cc2-465b-b01b-6f1e8142b94b-host\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.975071 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975008    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjthp\" (UniqueName: \"kubernetes.io/projected/68e2f225-4cc2-465b-b01b-6f1e8142b94b-kube-api-access-fjthp\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.975071 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-systemd-units\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.975251 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975098    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-systemd-units\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.975251 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975198    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-etc-selinux\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.975251 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-device-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.975251 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975247    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-run-netns\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975272    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-host\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975298    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-system-cni-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975328    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-run-netns\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-device-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-cnibin\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975381    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-netns\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975387    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-cnibin\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975379    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-etc-selinux\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975396    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-system-cni-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-netns\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.975456 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975449    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-ovn\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-sys\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975525    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975524    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-kubelet-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975580    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-ovn\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-node-log\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975635    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-node-log\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975695    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975735    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rdz4\" (UniqueName: \"kubernetes.io/projected/bf493cbb-ad21-4177-b194-d73d297a4cb0-kube-api-access-9rdz4\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975763    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-run\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975790    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975816    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-cni-multus\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-daemon-config\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975867    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-socket-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975886    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-cni-multus\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975901    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-tuning-conf-dir\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.976018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-systemd\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975954    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975984    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-ovnkube-config\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976010    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pfl8\" (UniqueName: \"kubernetes.io/projected/598105ef-f9a1-43c1-abf2-827929b9291b-kube-api-access-8pfl8\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-socket-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.975933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-run-systemd\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976037    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-os-release\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-socket-dir-parent\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-run-ovn-kubernetes\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-os-release\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976181    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-cni-bin\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976185    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-socket-dir-parent\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-cni-bin\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976220    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68e2f225-4cc2-465b-b01b-6f1e8142b94b-serviceca\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976246    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976301    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-cni-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.976881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976329    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-k8s-cni-cncf-io\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dws66\" (UniqueName: \"kubernetes.io/projected/2203e737-ecf1-46d7-81ea-be5ee9c0d476-kube-api-access-dws66\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qb252\" (UniqueName: \"kubernetes.io/projected/7b9c89ab-1a88-40b8-90d2-8c7230405e74-kube-api-access-qb252\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976418    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-modprobe-d\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976448    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-tuned\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976473    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5719ef18-02ca-43e8-ac4a-342d57f1f6cc-konnectivity-ca\") pod \"konnectivity-agent-9j6k6\" (UID: \"5719ef18-02ca-43e8-ac4a-342d57f1f6cc\") " pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976500    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-env-overrides\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976535    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-cni-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-cnibin\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976542    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-k8s-cni-cncf-io\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976566    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-daemon-config\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976622    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-cnibin\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976684    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-ovnkube-config\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976742    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mffkp\" (UniqueName: \"kubernetes.io/projected/bcd6660e-bbc4-47a2-94aa-d65b15d96247-kube-api-access-mffkp\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976812    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2203e737-ecf1-46d7-81ea-be5ee9c0d476-cni-binary-copy\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976840    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-registration-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976888    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-os-release\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.977604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976996    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/68e2f225-4cc2-465b-b01b-6f1e8142b94b-serviceca\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976962    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-os-release\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-env-overrides\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977036    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-hostroot\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-hostroot\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977073    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-conf-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-var-lib-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.976902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-registration-dir\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-cni-bin\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvdfg\" (UniqueName: \"kubernetes.io/projected/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-kube-api-access-jvdfg\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-var-lib-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977188    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-multus-conf-dir\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977240    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-cni-bin\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977257    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/2203e737-ecf1-46d7-81ea-be5ee9c0d476-cni-binary-copy\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-sys-fs\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977307    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-kubelet\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-cni-binary-copy\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977353    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-slash\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.978362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977383    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-slash\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/7b9c89ab-1a88-40b8-90d2-8c7230405e74-sys-fs\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977400    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-kubelet\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977406    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-lib-modules\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977453    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d8467c-ac8e-4e58-87c2-bf13e6faa093-tmp\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977487    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-dbus\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977545    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977570    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-systemd\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977600    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977621    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-etc-kubernetes\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977644    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-hosts-file\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977668    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-ovnkube-script-lib\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977692    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-etc-kubernetes\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysconfig\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.979143 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977732    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-hosts-file\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-kubernetes\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977837    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysctl-d\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977864    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-92mtb\" (UniqueName: \"kubernetes.io/projected/35d8467c-ac8e-4e58-87c2-bf13e6faa093-kube-api-access-92mtb\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-multus-certs\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977932    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-etc-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-etc-openvswitch\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.977993    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-run-multus-certs\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978019    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978027    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-system-cni-dir\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978054    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysctl-conf\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/bf493cbb-ad21-4177-b194-d73d297a4cb0-system-cni-dir\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978096    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-kubelet-config\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978150    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-kubelet\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-tmp-dir\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978228    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sglvr\" (UniqueName: \"kubernetes.io/projected/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-kube-api-access-sglvr\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978245    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2203e737-ecf1-46d7-81ea-be5ee9c0d476-host-var-lib-kubelet\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.979786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-log-socket\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-log-socket\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978339    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/bf493cbb-ad21-4177-b194-d73d297a4cb0-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978343    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-cni-netd\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978377    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/598105ef-f9a1-43c1-abf2-827929b9291b-host-cni-netd\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978391    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-var-lib-kubelet\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978420    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/598105ef-f9a1-43c1-abf2-827929b9291b-ovn-node-metrics-cert\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978443    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-iptables-alerter-script\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978485    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-host-slash\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978509    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5719ef18-02ca-43e8-ac4a-342d57f1f6cc-agent-certs\") pod \"konnectivity-agent-9j6k6\" (UID: \"5719ef18-02ca-43e8-ac4a-342d57f1f6cc\") " pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978528    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-tmp-dir\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/598105ef-f9a1-43c1-abf2-827929b9291b-ovnkube-script-lib\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.980367 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.978746    2561 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 31 14:48:25.982437 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.982417    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/598105ef-f9a1-43c1-abf2-827929b9291b-ovn-node-metrics-cert\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.986061 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.985979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjthp\" (UniqueName: \"kubernetes.io/projected/68e2f225-4cc2-465b-b01b-6f1e8142b94b-kube-api-access-fjthp\") pod \"node-ca-lz89t\" (UID: \"68e2f225-4cc2-465b-b01b-6f1e8142b94b\") " pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:25.988298 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.988274    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dws66\" (UniqueName: \"kubernetes.io/projected/2203e737-ecf1-46d7-81ea-be5ee9c0d476-kube-api-access-dws66\") pod \"multus-cx6kt\" (UID: \"2203e737-ecf1-46d7-81ea-be5ee9c0d476\") " pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:25.988729 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.988690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pfl8\" (UniqueName: \"kubernetes.io/projected/598105ef-f9a1-43c1-abf2-827929b9291b-kube-api-access-8pfl8\") pod \"ovnkube-node-k2dtc\" (UID: \"598105ef-f9a1-43c1-abf2-827929b9291b\") " pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:25.988822 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.988764    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qb252\" (UniqueName: \"kubernetes.io/projected/7b9c89ab-1a88-40b8-90d2-8c7230405e74-kube-api-access-qb252\") pod \"aws-ebs-csi-driver-node-frqgh\" (UID: \"7b9c89ab-1a88-40b8-90d2-8c7230405e74\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:25.989792 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.989771    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sglvr\" (UniqueName: \"kubernetes.io/projected/98896ea7-ed6d-451c-a61b-e74ab2b1cdf9-kube-api-access-sglvr\") pod \"node-resolver-wr8xz\" (UID: \"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9\") " pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:25.990058 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:25.990037    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rdz4\" (UniqueName: \"kubernetes.io/projected/bf493cbb-ad21-4177-b194-d73d297a4cb0-kube-api-access-9rdz4\") pod \"multus-additional-cni-plugins-7ctdq\" (UID: \"bf493cbb-ad21-4177-b194-d73d297a4cb0\") " pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:26.078999 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.078956    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysconfig\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.078999 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079003    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-kubernetes\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079028    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysctl-d\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-92mtb\" (UniqueName: \"kubernetes.io/projected/35d8467c-ac8e-4e58-87c2-bf13e6faa093-kube-api-access-92mtb\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079070    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysconfig\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079084    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysctl-conf\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079125    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-kubelet-config\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079133    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-kubernetes\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079170    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-var-lib-kubelet\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079201    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-iptables-alerter-script\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079215    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-kubelet-config\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079224    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-host-slash\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:26.079240 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079234    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysctl-conf\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079248    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5719ef18-02ca-43e8-ac4a-342d57f1f6cc-agent-certs\") pod \"konnectivity-agent-9j6k6\" (UID: \"5719ef18-02ca-43e8-ac4a-342d57f1f6cc\") " pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079278    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-host\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-host-slash\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079296    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-var-lib-kubelet\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079308    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-sys\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079334    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-run\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079346    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-host\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079359    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079275    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-sysctl-d\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079401    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079406    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-run\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-sys\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079445    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-modprobe-d\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079472    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-tuned\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079490    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5719ef18-02ca-43e8-ac4a-342d57f1f6cc-konnectivity-ca\") pod \"konnectivity-agent-9j6k6\" (UID: \"5719ef18-02ca-43e8-ac4a-342d57f1f6cc\") " pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079516    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mffkp\" (UniqueName: \"kubernetes.io/projected/bcd6660e-bbc4-47a2-94aa-d65b15d96247-kube-api-access-mffkp\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079548    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvdfg\" (UniqueName: \"kubernetes.io/projected/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-kube-api-access-jvdfg\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:26.079669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079578    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-lib-modules\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079601    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d8467c-ac8e-4e58-87c2-bf13e6faa093-tmp\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.079491    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079622    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-dbus\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079647    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-systemd\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079703    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-systemd\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.079708    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret podName:7236d2d7-d084-40e7-9b8d-cd72aa4243b7 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.579664139 +0000 UTC m=+3.099088174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret") pod "global-pull-secret-syncer-gfczg" (UID: "7236d2d7-d084-40e7-9b8d-cd72aa4243b7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079734    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079579    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-modprobe-d\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.079576    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079825    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-iptables-alerter-script\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.079843    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs podName:bcd6660e-bbc4-47a2-94aa-d65b15d96247 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.57982718 +0000 UTC m=+3.099251204 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs") pod "network-metrics-daemon-qpj79" (UID: "bcd6660e-bbc4-47a2-94aa-d65b15d96247") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.079948    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/35d8467c-ac8e-4e58-87c2-bf13e6faa093-lib-modules\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.080029    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5719ef18-02ca-43e8-ac4a-342d57f1f6cc-konnectivity-ca\") pod \"konnectivity-agent-9j6k6\" (UID: \"5719ef18-02ca-43e8-ac4a-342d57f1f6cc\") " pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:26.080383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.080097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-dbus\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:26.081719 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.081698    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5719ef18-02ca-43e8-ac4a-342d57f1f6cc-agent-certs\") pod \"konnectivity-agent-9j6k6\" (UID: \"5719ef18-02ca-43e8-ac4a-342d57f1f6cc\") " pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:26.081810 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.081794    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/35d8467c-ac8e-4e58-87c2-bf13e6faa093-tmp\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.082174 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.082146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/35d8467c-ac8e-4e58-87c2-bf13e6faa093-etc-tuned\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.091869 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.091848    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:26.091869 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.091870    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:26.092029 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.091881    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r4q4l for pod openshift-network-diagnostics/network-check-target-ch9qv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.092029 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.091934    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l podName:73a4030c-d937-4fde-b65c-e026541f6cbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:26.591918457 +0000 UTC m=+3.111342479 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-r4q4l" (UniqueName: "kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l") pod "network-check-target-ch9qv" (UID: "73a4030c-d937-4fde-b65c-e026541f6cbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.093377 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.093355    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvdfg\" (UniqueName: \"kubernetes.io/projected/4bd381bb-8a21-4c36-a8ce-bbf3ecebab43-kube-api-access-jvdfg\") pod \"iptables-alerter-9gzdg\" (UID: \"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43\") " pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:26.093489 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.093357    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-92mtb\" (UniqueName: \"kubernetes.io/projected/35d8467c-ac8e-4e58-87c2-bf13e6faa093-kube-api-access-92mtb\") pod \"tuned-dgg8r\" (UID: \"35d8467c-ac8e-4e58-87c2-bf13e6faa093\") " pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.093874 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.093856    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mffkp\" (UniqueName: \"kubernetes.io/projected/bcd6660e-bbc4-47a2-94aa-d65b15d96247-kube-api-access-mffkp\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:26.170166 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.170117    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cx6kt"
Jul 31 14:48:26.182329 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.182263    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh"
Jul 31 14:48:26.190983 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.190963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-wr8xz"
Jul 31 14:48:26.191073 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:26.191044    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b9c89ab_1a88_40b8_90d2_8c7230405e74.slice/crio-70742a548e96a1253ae6addf0224cd2e066c06d7c269e6568cb355cf45db7182 WatchSource:0}: Error finding container 70742a548e96a1253ae6addf0224cd2e066c06d7c269e6568cb355cf45db7182: Status 404 returned error can't find the container with id 70742a548e96a1253ae6addf0224cd2e066c06d7c269e6568cb355cf45db7182
Jul 31 14:48:26.196342 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.196317    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lz89t"
Jul 31 14:48:26.202442 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.202420    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:26.202718 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:26.202691    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98896ea7_ed6d_451c_a61b_e74ab2b1cdf9.slice/crio-ab50acd40195ed44d178feddbd4e741fabe5dbac32d01f8ba88a5e89299f24e9 WatchSource:0}: Error finding container ab50acd40195ed44d178feddbd4e741fabe5dbac32d01f8ba88a5e89299f24e9: Status 404 returned error can't find the container with id ab50acd40195ed44d178feddbd4e741fabe5dbac32d01f8ba88a5e89299f24e9
Jul 31 14:48:26.208847 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.208819    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-7ctdq"
Jul 31 14:48:26.210870 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:26.210839    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598105ef_f9a1_43c1_abf2_827929b9291b.slice/crio-260edcf4dafb8a10a810f4e0f1b7e80d01c9b7d6f6e2b7d523af963b56ab6cbf WatchSource:0}: Error finding container 260edcf4dafb8a10a810f4e0f1b7e80d01c9b7d6f6e2b7d523af963b56ab6cbf: Status 404 returned error can't find the container with id 260edcf4dafb8a10a810f4e0f1b7e80d01c9b7d6f6e2b7d523af963b56ab6cbf
Jul 31 14:48:26.215752 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.215702    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:26.217828 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:26.217802    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf493cbb_ad21_4177_b194_d73d297a4cb0.slice/crio-1d19b00a77a481af26ebe072babe0ce2e38f4bd140366d98ea385e74c3a91361 WatchSource:0}: Error finding container 1d19b00a77a481af26ebe072babe0ce2e38f4bd140366d98ea385e74c3a91361: Status 404 returned error can't find the container with id 1d19b00a77a481af26ebe072babe0ce2e38f4bd140366d98ea385e74c3a91361
Jul 31 14:48:26.221664 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.221587    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-dgg8r"
Jul 31 14:48:26.226751 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:26.226729    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5719ef18_02ca_43e8_ac4a_342d57f1f6cc.slice/crio-8675cc68ab045d8e120c77850017de8598603f9b39b419770174e36bd8eec7df WatchSource:0}: Error finding container 8675cc68ab045d8e120c77850017de8598603f9b39b419770174e36bd8eec7df: Status 404 returned error can't find the container with id 8675cc68ab045d8e120c77850017de8598603f9b39b419770174e36bd8eec7df
Jul 31 14:48:26.228668 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.228642    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-9gzdg"
Jul 31 14:48:26.230857 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:26.230834    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35d8467c_ac8e_4e58_87c2_bf13e6faa093.slice/crio-fe8edf7c2adc9de5009e05a173043b9457104b0e95b9b0f17cd0b706f05143a8 WatchSource:0}: Error finding container fe8edf7c2adc9de5009e05a173043b9457104b0e95b9b0f17cd0b706f05143a8: Status 404 returned error can't find the container with id fe8edf7c2adc9de5009e05a173043b9457104b0e95b9b0f17cd0b706f05143a8
Jul 31 14:48:26.237196 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:26.237173    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bd381bb_8a21_4c36_a8ce_bbf3ecebab43.slice/crio-5e46f70f02b5b5607db245d9618233ea8eb638fe067330e4d3af2c0553abe11b WatchSource:0}: Error finding container 5e46f70f02b5b5607db245d9618233ea8eb638fe067330e4d3af2c0553abe11b: Status 404 returned error can't find the container with id 5e46f70f02b5b5607db245d9618233ea8eb638fe067330e4d3af2c0553abe11b
Jul 31 14:48:26.354506 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.354296    2561 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 31 14:48:26.584491 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.584242    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:26.584491 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.584300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:26.584491 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.584361    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.584491 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.584409    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.584491 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.584439    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret podName:7236d2d7-d084-40e7-9b8d-cd72aa4243b7 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:27.584417987 +0000 UTC m=+4.103842027 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret") pod "global-pull-secret-syncer-gfczg" (UID: "7236d2d7-d084-40e7-9b8d-cd72aa4243b7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:26.584491 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.584463    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs podName:bcd6660e-bbc4-47a2-94aa-d65b15d96247 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:27.584446267 +0000 UTC m=+4.103870301 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs") pod "network-metrics-daemon-qpj79" (UID: "bcd6660e-bbc4-47a2-94aa-d65b15d96247") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:26.686121 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.685380    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:26.686121 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.685626    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:26.686121 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.685645    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:26.686121 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.685658    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r4q4l for pod openshift-network-diagnostics/network-check-target-ch9qv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.686121 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.685721    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l podName:73a4030c-d937-4fde-b65c-e026541f6cbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:27.685701719 +0000 UTC m=+4.205125740 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4q4l" (UniqueName: "kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l") pod "network-check-target-ch9qv" (UID: "73a4030c-d937-4fde-b65c-e026541f6cbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:26.916967 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.916918    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:25 +0000 UTC" deadline="2028-03-28 10:33:04.864141482 +0000 UTC"
Jul 31 14:48:26.916967 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.916965    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14539h44m37.947180959s"
Jul 31 14:48:26.956592 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.956516    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:26.956745 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:26.956645    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:26.977685 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.977386    2561 generic.go:356] "Generic (PLEG): container finished" podID="085be5e42d0150ff2c7d62ba258a4275" containerID="48a9cafa64fa79b2c72bcf1f8491f51b4eb6cee1ab0a0a7ce8f9613e9dcbc788" exitCode=0
Jul 31 14:48:26.977685 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.977486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal" event={"ID":"085be5e42d0150ff2c7d62ba258a4275","Type":"ContainerDied","Data":"48a9cafa64fa79b2c72bcf1f8491f51b4eb6cee1ab0a0a7ce8f9613e9dcbc788"}
Jul 31 14:48:26.990004 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.989973    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9gzdg" event={"ID":"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43","Type":"ContainerStarted","Data":"5e46f70f02b5b5607db245d9618233ea8eb638fe067330e4d3af2c0553abe11b"}
Jul 31 14:48:26.999400 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:26.999372    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dgg8r" event={"ID":"35d8467c-ac8e-4e58-87c2-bf13e6faa093","Type":"ContainerStarted","Data":"fe8edf7c2adc9de5009e05a173043b9457104b0e95b9b0f17cd0b706f05143a8"}
Jul 31 14:48:27.006421 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.006392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9j6k6" event={"ID":"5719ef18-02ca-43e8-ac4a-342d57f1f6cc","Type":"ContainerStarted","Data":"8675cc68ab045d8e120c77850017de8598603f9b39b419770174e36bd8eec7df"}
Jul 31 14:48:27.015990 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.015759    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"260edcf4dafb8a10a810f4e0f1b7e80d01c9b7d6f6e2b7d523af963b56ab6cbf"}
Jul 31 14:48:27.021463 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.021435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wr8xz" event={"ID":"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9","Type":"ContainerStarted","Data":"ab50acd40195ed44d178feddbd4e741fabe5dbac32d01f8ba88a5e89299f24e9"}
Jul 31 14:48:27.028201 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.028173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh" event={"ID":"7b9c89ab-1a88-40b8-90d2-8c7230405e74","Type":"ContainerStarted","Data":"70742a548e96a1253ae6addf0224cd2e066c06d7c269e6568cb355cf45db7182"}
Jul 31 14:48:27.049345 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.049312    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerStarted","Data":"1d19b00a77a481af26ebe072babe0ce2e38f4bd140366d98ea385e74c3a91361"}
Jul 31 14:48:27.079064 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.078936    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lz89t" event={"ID":"68e2f225-4cc2-465b-b01b-6f1e8142b94b","Type":"ContainerStarted","Data":"f73829865561895a070ff81034fa3b926466c9074beb85af5fd8c1d81f5675fe"}
Jul 31 14:48:27.114858 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.114770    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cx6kt" event={"ID":"2203e737-ecf1-46d7-81ea-be5ee9c0d476","Type":"ContainerStarted","Data":"67b0968fc0da3c96ff164db0f3af5e0ff8713243a5a532fed5221eeae6b37543"}
Jul 31 14:48:27.592119 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.592079    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:27.592304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.592143    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:27.592377 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.592321    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:27.592426 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.592388    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs podName:bcd6660e-bbc4-47a2-94aa-d65b15d96247 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:29.592367851 +0000 UTC m=+6.111791872 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs") pod "network-metrics-daemon-qpj79" (UID: "bcd6660e-bbc4-47a2-94aa-d65b15d96247") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:27.592493 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.592466    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:27.592538 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.592501    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret podName:7236d2d7-d084-40e7-9b8d-cd72aa4243b7 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:29.592489414 +0000 UTC m=+6.111913430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret") pod "global-pull-secret-syncer-gfczg" (UID: "7236d2d7-d084-40e7-9b8d-cd72aa4243b7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:27.692920 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.692837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:27.693092 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.693072    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:27.693145 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.693097    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:27.693145 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.693113    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r4q4l for pod openshift-network-diagnostics/network-check-target-ch9qv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:27.693293 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.693187    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l podName:73a4030c-d937-4fde-b65c-e026541f6cbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:29.693168441 +0000 UTC m=+6.212592477 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4q4l" (UniqueName: "kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l") pod "network-check-target-ch9qv" (UID: "73a4030c-d937-4fde-b65c-e026541f6cbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:27.917515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.917420    2561 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-30 14:43:25 +0000 UTC" deadline="2028-02-16 08:48:43.234970696 +0000 UTC"
Jul 31 14:48:27.917515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.917479    2561 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13554h0m15.317495839s"
Jul 31 14:48:27.957661 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.956790    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:27.957661 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.956939    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:27.957661 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:27.957369    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:27.957661 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:27.957470    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:28.130600 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:28.130481    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal" event={"ID":"085be5e42d0150ff2c7d62ba258a4275","Type":"ContainerStarted","Data":"3ac1a7060dff2008ebd12391829dda1a82a9723a655908f300c93402b1a0a4d6"}
Jul 31 14:48:28.955672 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:28.955644    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:28.956107 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:28.955774    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:29.610355 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:29.610056    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:29.610355 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:29.610118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:29.610355 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.610244    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:29.610355 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.610264    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:29.610355 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.610312    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs podName:bcd6660e-bbc4-47a2-94aa-d65b15d96247 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:33.610293613 +0000 UTC m=+10.129717644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs") pod "network-metrics-daemon-qpj79" (UID: "bcd6660e-bbc4-47a2-94aa-d65b15d96247") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:29.610355 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.610328    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret podName:7236d2d7-d084-40e7-9b8d-cd72aa4243b7 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:33.610320028 +0000 UTC m=+10.129744044 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret") pod "global-pull-secret-syncer-gfczg" (UID: "7236d2d7-d084-40e7-9b8d-cd72aa4243b7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:29.711496 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:29.710741    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:29.711496 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.710957    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:29.711496 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.710982    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:29.711496 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.710998    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r4q4l for pod openshift-network-diagnostics/network-check-target-ch9qv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:29.711496 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.711063    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l podName:73a4030c-d937-4fde-b65c-e026541f6cbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:33.711043661 +0000 UTC m=+10.230467699 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4q4l" (UniqueName: "kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l") pod "network-check-target-ch9qv" (UID: "73a4030c-d937-4fde-b65c-e026541f6cbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:29.958891 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:29.958203    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:29.958891 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.958391    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:29.958891 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:29.958786    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:29.959461 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:29.958900    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:30.955976 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:30.955944    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:30.956168 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:30.956080    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:31.958164 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:31.958120    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:31.958618 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:31.958292    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:31.958618 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:31.958362    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:31.958618 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:31.958433    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:32.956278 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:32.956248    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:32.956452 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:32.956364    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:33.649107 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:33.648571    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:33.649107 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:33.648631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:33.649107 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.648809    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:33.649107 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.648885    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs podName:bcd6660e-bbc4-47a2-94aa-d65b15d96247 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.648863138 +0000 UTC m=+18.168287155 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs") pod "network-metrics-daemon-qpj79" (UID: "bcd6660e-bbc4-47a2-94aa-d65b15d96247") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:33.649107 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.648955    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:33.649107 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.649000    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret podName:7236d2d7-d084-40e7-9b8d-cd72aa4243b7 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.648986769 +0000 UTC m=+18.168410799 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret") pod "global-pull-secret-syncer-gfczg" (UID: "7236d2d7-d084-40e7-9b8d-cd72aa4243b7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:33.749384 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:33.749344    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:33.749570 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.749483    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:33.749570 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.749503    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:33.749570 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.749516    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r4q4l for pod openshift-network-diagnostics/network-check-target-ch9qv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:33.749754 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.749583    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l podName:73a4030c-d937-4fde-b65c-e026541f6cbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:41.749564034 +0000 UTC m=+18.268988067 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4q4l" (UniqueName: "kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l") pod "network-check-target-ch9qv" (UID: "73a4030c-d937-4fde-b65c-e026541f6cbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:33.956276 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:33.956189    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:33.956424 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.956335    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:33.957971 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:33.956732    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:33.957971 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:33.956866    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:34.957173 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:34.956602    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:34.957173 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:34.956743    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:35.956012 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:35.955969    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:35.956223 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:35.955973    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:35.956223 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:35.956113    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:35.956223 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:35.956174    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:36.955576 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:36.955540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:36.955959 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:36.955651    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:37.956663 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:37.956625    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:37.957048 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:37.956682    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:37.957048 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:37.956769    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:37.957048 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:37.956937    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:38.956165 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:38.956115    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:38.956350 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:38.956266    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:39.955864 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:39.955821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:39.955864 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:39.955853    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:39.956396 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:39.955991    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:39.956396 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:39.956131    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:40.956115 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:40.956077    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:40.956719 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:40.956226    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:41.711983 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:41.711953    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:41.712233 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:41.711993    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:41.712233 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.712177    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:41.712233 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.712213    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:41.712394 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.712266    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret podName:7236d2d7-d084-40e7-9b8d-cd72aa4243b7 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.712243873 +0000 UTC m=+34.231667903 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret") pod "global-pull-secret-syncer-gfczg" (UID: "7236d2d7-d084-40e7-9b8d-cd72aa4243b7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:41.712394 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.712288    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs podName:bcd6660e-bbc4-47a2-94aa-d65b15d96247 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.71227665 +0000 UTC m=+34.231700678 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs") pod "network-metrics-daemon-qpj79" (UID: "bcd6660e-bbc4-47a2-94aa-d65b15d96247") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:41.812421 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:41.812390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:41.812581 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.812534    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:41.812581 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.812554    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:41.812581 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.812566    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r4q4l for pod openshift-network-diagnostics/network-check-target-ch9qv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:41.812705 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.812627    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l podName:73a4030c-d937-4fde-b65c-e026541f6cbe nodeName:}" failed. No retries permitted until 2026-07-31 14:48:57.812611152 +0000 UTC m=+34.332035183 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4q4l" (UniqueName: "kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l") pod "network-check-target-ch9qv" (UID: "73a4030c-d937-4fde-b65c-e026541f6cbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:41.956135 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:41.956102    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:41.956583 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:41.956102    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:41.956583 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.956281    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:41.956583 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:41.956405    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:42.955671 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:42.955640    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:42.955866 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:42.955764    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:43.957535 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:43.957502    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:43.957962 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:43.957616    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:43.957962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:43.957672    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:43.957962 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:43.957792    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:44.956360 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:44.956325    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:44.956581 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:44.956474    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:45.957061 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:45.956733    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:45.957061 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:45.956881    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:45.957061 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:45.956940    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:45.957061 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:45.957031    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:46.170545 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.170195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cx6kt" event={"ID":"2203e737-ecf1-46d7-81ea-be5ee9c0d476","Type":"ContainerStarted","Data":"d08496c1bc00ee4976451e39610d7a1476d07a43cb987932ab1e750beabfb73b"}
Jul 31 14:48:46.174819 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.174787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal" event={"ID":"645b936c36ac9f10f261c827f21930d7","Type":"ContainerStarted","Data":"f9fc133164ed98e033af654393dc11b70d9467e3422862846fedbb80302587ab"}
Jul 31 14:48:46.178243 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.178212    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-dgg8r" event={"ID":"35d8467c-ac8e-4e58-87c2-bf13e6faa093","Type":"ContainerStarted","Data":"7464ef32e483117597abb4b62e2f984c1f5116d69d6387fc1bafa5e7317ecb70"}
Jul 31 14:48:46.180095 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.180050    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9j6k6" event={"ID":"5719ef18-02ca-43e8-ac4a-342d57f1f6cc","Type":"ContainerStarted","Data":"12fa1a1dd600ad1de8c8891f9f8ffb7d2a6cff373086697f4a7fc936ba166391"}
Jul 31 14:48:46.181816 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.181795    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"236dd6f52f9dfa77d59c100460a9eff86bc7234032380f3a8173705825df9b26"}
Jul 31 14:48:46.183331 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.183306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-wr8xz" event={"ID":"98896ea7-ed6d-451c-a61b-e74ab2b1cdf9","Type":"ContainerStarted","Data":"55435d250d8a1b103e61225411792f7e75525cd1a80fd1d8f14090d8212a6e5b"}
Jul 31 14:48:46.185087 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.185054    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh" event={"ID":"7b9c89ab-1a88-40b8-90d2-8c7230405e74","Type":"ContainerStarted","Data":"956c29dcd680a4034d5ff856e9bb1a77ba90bfd361118302f4d4c8a421f829e7"}
Jul 31 14:48:46.190351 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.190329    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerStarted","Data":"67ba5bf8211977140297080aa8bd039d8bb66400ab1f7aa24ec2ccd8e0cf3073"}
Jul 31 14:48:46.193321 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.193292    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lz89t" event={"ID":"68e2f225-4cc2-465b-b01b-6f1e8142b94b","Type":"ContainerStarted","Data":"5c97e68fd058955f720d21e204f8d7d907bc280be43d22a6ba00b23f26da644b"}
Jul 31 14:48:46.196020 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.195968    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-192.ec2.internal" podStartSLOduration=21.195951472 podStartE2EDuration="21.195951472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:28.151316141 +0000 UTC m=+4.670740175" watchObservedRunningTime="2026-07-31 14:48:46.195951472 +0000 UTC m=+22.715375511"
Jul 31 14:48:46.196248 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.196216    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cx6kt" podStartSLOduration=2.403397011 podStartE2EDuration="22.196205267s" totalImagesPullingTime="19.792808256s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.195399328 +0000 UTC m=+22.714823366" watchObservedRunningTime="2026-07-31 14:48:46.196205267 +0000 UTC m=+22.715629308"
Jul 31 14:48:46.236241 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.236187    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-192.ec2.internal" podStartSLOduration=21.236167681 podStartE2EDuration="21.236167681s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.217016087 +0000 UTC m=+22.736440137" watchObservedRunningTime="2026-07-31 14:48:46.236167681 +0000 UTC m=+22.755591711"
Jul 31 14:48:46.265560 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.265511    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-wr8xz" podStartSLOduration=2.564710753 podStartE2EDuration="22.265496419s" totalImagesPullingTime="19.700785666s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.238800443 +0000 UTC m=+22.758224482" watchObservedRunningTime="2026-07-31 14:48:46.265496419 +0000 UTC m=+22.784920456"
Jul 31 14:48:46.265685 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.265592    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lz89t" podStartSLOduration=2.5595382989999997 podStartE2EDuration="22.265587386s" totalImagesPullingTime="19.706049087s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.265015427 +0000 UTC m=+22.784439459" watchObservedRunningTime="2026-07-31 14:48:46.265587386 +0000 UTC m=+22.785011423"
Jul 31 14:48:46.325542 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.325497    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9j6k6" podStartSLOduration=2.639851923 podStartE2EDuration="22.325480951s" totalImagesPullingTime="19.685629028s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.316772616 +0000 UTC m=+22.836196655" watchObservedRunningTime="2026-07-31 14:48:46.325480951 +0000 UTC m=+22.844904988"
Jul 31 14:48:46.956451 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:46.956383    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:46.956609 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:46.956514    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:47.200229 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.200183    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"3c49038b68840546988b5cac3e6894755510521b047a69f0e4ff7844e45708eb"}
Jul 31 14:48:47.201360 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.200321    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"4d11c57743c90c5005b2284d43759e7655e3ff488e2883fe5be33f57e6871fb9"}
Jul 31 14:48:47.201360 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.200363    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"e8ed580e980565a000a0b4bc13013d0e36e925b4de80b1722d4e7b4c0e111d3b"}
Jul 31 14:48:47.201360 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.200404    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"58d4bfbff3cd88a43308e36e9037c2679df08ca4ce887f14b69cdc044fb6bb45"}
Jul 31 14:48:47.201360 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.200454    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"0d277050df803e4da0dad56abb9a6b3e5664a12aad66e26e97b6bf7fc6a5157a"}
Jul 31 14:48:47.202034 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.202006    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf493cbb-ad21-4177-b194-d73d297a4cb0" containerID="67ba5bf8211977140297080aa8bd039d8bb66400ab1f7aa24ec2ccd8e0cf3073" exitCode=0
Jul 31 14:48:47.202193 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.202098    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerDied","Data":"67ba5bf8211977140297080aa8bd039d8bb66400ab1f7aa24ec2ccd8e0cf3073"}
Jul 31 14:48:47.406249 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.406069    2561 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 31 14:48:47.941738 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.941634    2561 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-31T14:48:47.406087773Z","UUID":"a7466b43-bfd2-4187-869b-b95e4bd7002c","Handler":null,"Name":"","Endpoint":""}
Jul 31 14:48:47.943546 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.943526    2561 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 31 14:48:47.943687 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.943564    2561 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 31 14:48:47.956248 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.956208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:47.956424 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:47.956208    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:47.956424 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:47.956349    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:47.956557 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:47.956426    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:48.206362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:48.206272    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh" event={"ID":"7b9c89ab-1a88-40b8-90d2-8c7230405e74","Type":"ContainerStarted","Data":"fcac612a1af4a60d81656a4414cc91fff086201569fca8fbdc79eaaf9e217810"}
Jul 31 14:48:48.207891 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:48.207856    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-9gzdg" event={"ID":"4bd381bb-8a21-4c36-a8ce-bbf3ecebab43","Type":"ContainerStarted","Data":"e10b921681217b40b5405a1bea202775fbc6b23179b8c09a2a81514842f85762"}
Jul 31 14:48:48.229976 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:48.229917    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-9gzdg" podStartSLOduration=4.550590531 podStartE2EDuration="24.229899171s" totalImagesPullingTime="19.67930864s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:48.229845201 +0000 UTC m=+24.749269240" watchObservedRunningTime="2026-07-31 14:48:48.229899171 +0000 UTC m=+24.749323208"
Jul 31 14:48:48.230383 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:48.230357    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-dgg8r" podStartSLOduration=4.557784102 podStartE2EDuration="24.230348856s" totalImagesPullingTime="19.672564754s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:46.340633612 +0000 UTC m=+22.860057641" watchObservedRunningTime="2026-07-31 14:48:48.230348856 +0000 UTC m=+24.749772894"
Jul 31 14:48:48.956370 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:48.956277    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:48.956541 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:48.956444    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:49.027205 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:49.027165    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:49.956338 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:49.956310    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:49.956338 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:49.956322    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:49.956897 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:49.956449    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:49.956897 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:49.956576    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:50.220187 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:50.219872    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"200d449c16c32649487a715e9d69866d944999f5e0b655b9a3cffe07547ed775"}
Jul 31 14:48:50.500496 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:50.500426    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:50.501042 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:50.501023    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:50.956137 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:50.956112    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:50.956277 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:50.956231    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:51.223576 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:51.223495    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf493cbb-ad21-4177-b194-d73d297a4cb0" containerID="de969a8d4b495090d16d8c479732ea761900d911198f157bb42cb2d1032dd37f" exitCode=0
Jul 31 14:48:51.224355 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:51.223587    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerDied","Data":"de969a8d4b495090d16d8c479732ea761900d911198f157bb42cb2d1032dd37f"}
Jul 31 14:48:51.225789 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:51.225766    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh" event={"ID":"7b9c89ab-1a88-40b8-90d2-8c7230405e74","Type":"ContainerStarted","Data":"1d54c09dfcb6035ab3eccefefe4aca9b0e8f449b01051d54d99a4599042e3c53"}
Jul 31 14:48:51.226364 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:51.226346    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9j6k6"
Jul 31 14:48:51.269733 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:51.269682    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-frqgh" podStartSLOduration=2.95540678 podStartE2EDuration="27.269667848s" totalImagesPullingTime="24.314261068s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:51.269258261 +0000 UTC m=+27.788682298" watchObservedRunningTime="2026-07-31 14:48:51.269667848 +0000 UTC m=+27.789091933"
Jul 31 14:48:51.956503 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:51.956460    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:51.956713 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:51.956455    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:51.956713 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:51.956599    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:51.956713 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:51.956668    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:52.956059 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:52.955864    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:52.956685 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:52.956143    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:53.232593 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.232506    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf493cbb-ad21-4177-b194-d73d297a4cb0" containerID="2aa8ab881baaba3e7e64eee592f59dabdf00dd0a612e49366c03fda72c030ca6" exitCode=0
Jul 31 14:48:53.232593 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.232569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerDied","Data":"2aa8ab881baaba3e7e64eee592f59dabdf00dd0a612e49366c03fda72c030ca6"}
Jul 31 14:48:53.236402 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.236376    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" event={"ID":"598105ef-f9a1-43c1-abf2-827929b9291b","Type":"ContainerStarted","Data":"752c30c6b825814313fa43d200e9bb68f1ddf6f121310e5a0841b52b73d98d6f"}
Jul 31 14:48:53.236668 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.236649    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:53.236738 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.236731    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:53.252372 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.252346    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:53.286221 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.286174    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc" podStartSLOduration=9.516373296 podStartE2EDuration="29.286138706s" totalImagesPullingTime="19.76976541s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:48:53.285184165 +0000 UTC m=+29.804608204" watchObservedRunningTime="2026-07-31 14:48:53.286138706 +0000 UTC m=+29.805562758"
Jul 31 14:48:53.956385 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.956348    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:53.956914 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:53.956463    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:53.956914 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:53.956513    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:53.956914 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:53.956618    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:54.239646 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:54.239619    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:54.260630 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:54.260600    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:48:54.595108 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:54.594835    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ch9qv"]
Jul 31 14:48:54.595281 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:54.595213    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:54.595348 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:54.595322    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:54.595787 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:54.595757    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gfczg"]
Jul 31 14:48:54.595913 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:54.595876    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:54.596011 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:54.595982    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:54.610303 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:54.610275    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qpj79"]
Jul 31 14:48:54.610471 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:54.610410    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:54.610551 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:54.610527    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:55.244460 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:55.244427    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf493cbb-ad21-4177-b194-d73d297a4cb0" containerID="9b98b5a3d071aa453e4f370b7fc71f3712cd9451e5891a6461aabf110b7bbd5c" exitCode=0
Jul 31 14:48:55.245043 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:55.244510    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerDied","Data":"9b98b5a3d071aa453e4f370b7fc71f3712cd9451e5891a6461aabf110b7bbd5c"}
Jul 31 14:48:55.956048 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:55.956009    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:55.956253 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:55.956018    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:55.956253 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:55.956171    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-ch9qv" podUID="73a4030c-d937-4fde-b65c-e026541f6cbe"
Jul 31 14:48:55.956253 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:55.956234    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gfczg" podUID="7236d2d7-d084-40e7-9b8d-cd72aa4243b7"
Jul 31 14:48:56.956510 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:56.956469    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:56.957118 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:56.956620    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qpj79" podUID="bcd6660e-bbc4-47a2-94aa-d65b15d96247"
Jul 31 14:48:57.727105 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.727026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:57.727105 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.727072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:57.727343 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:57.727197    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:57.727343 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:57.727260    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs podName:bcd6660e-bbc4-47a2-94aa-d65b15d96247 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.727242146 +0000 UTC m=+66.246666162 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs") pod "network-metrics-daemon-qpj79" (UID: "bcd6660e-bbc4-47a2-94aa-d65b15d96247") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 31 14:48:57.727343 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:57.727198    2561 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:57.727343 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:57.727336    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret podName:7236d2d7-d084-40e7-9b8d-cd72aa4243b7 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.727320213 +0000 UTC m=+66.246744234 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret") pod "global-pull-secret-syncer-gfczg" (UID: "7236d2d7-d084-40e7-9b8d-cd72aa4243b7") : object "kube-system"/"original-pull-secret" not registered
Jul 31 14:48:57.801348 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.801316    2561 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-192.ec2.internal" event="NodeReady"
Jul 31 14:48:57.801516 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.801492    2561 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 31 14:48:57.827915 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.827881    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:57.828072 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:57.828009    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 31 14:48:57.828072 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:57.828027    2561 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 31 14:48:57.828072 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:57.828036    2561 projected.go:196] Error preparing data for projected volume kube-api-access-r4q4l for pod openshift-network-diagnostics/network-check-target-ch9qv: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:57.828197 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:57.828089    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l podName:73a4030c-d937-4fde-b65c-e026541f6cbe nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.828071965 +0000 UTC m=+66.347495984 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-r4q4l" (UniqueName: "kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l") pod "network-check-target-ch9qv" (UID: "73a4030c-d937-4fde-b65c-e026541f6cbe") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 31 14:48:57.890949 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.890911    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-hcnkz"]
Jul 31 14:48:57.914219 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.914190    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-mwkpt"]
Jul 31 14:48:57.914399 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.914299    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:57.918000 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.917904    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 31 14:48:57.918000 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.917923    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 31 14:48:57.918414 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.918396    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-plcv4\""
Jul 31 14:48:57.928133 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.928114    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hcnkz"]
Jul 31 14:48:57.928255 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.928167    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mwkpt"]
Jul 31 14:48:57.928312 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.928251    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mwkpt"
Jul 31 14:48:57.931938 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.931733    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 31 14:48:57.931938 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.931817    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 31 14:48:57.931938 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.931882    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 31 14:48:57.932147 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.931987    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p85t9\""
Jul 31 14:48:57.955948 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.955914    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:48:57.956141 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.955958    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:48:57.960090 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.960061    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wtnft\""
Jul 31 14:48:57.960090 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.960077    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 31 14:48:57.960696 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.960141    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:48:57.960696 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:57.960175    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 31 14:48:58.029085 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.028981    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.029085 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.029051    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slvf2\" (UniqueName: \"kubernetes.io/projected/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-kube-api-access-slvf2\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.029342 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.029114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjkf4\" (UniqueName: \"kubernetes.io/projected/27804094-6280-4f24-a5c7-0a0dc5ff6561-kube-api-access-mjkf4\") pod \"ingress-canary-mwkpt\" (UID: \"27804094-6280-4f24-a5c7-0a0dc5ff6561\") " pod="openshift-ingress-canary/ingress-canary-mwkpt"
Jul 31 14:48:58.029342 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.029178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-tmp-dir\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.029342 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.029267    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27804094-6280-4f24-a5c7-0a0dc5ff6561-cert\") pod \"ingress-canary-mwkpt\" (UID: \"27804094-6280-4f24-a5c7-0a0dc5ff6561\") " pod="openshift-ingress-canary/ingress-canary-mwkpt"
Jul 31 14:48:58.029342 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.029319    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-config-volume\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.130404 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.130361    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.130590 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.130426    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slvf2\" (UniqueName: \"kubernetes.io/projected/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-kube-api-access-slvf2\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.130590 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.130459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjkf4\" (UniqueName: \"kubernetes.io/projected/27804094-6280-4f24-a5c7-0a0dc5ff6561-kube-api-access-mjkf4\") pod \"ingress-canary-mwkpt\" (UID: \"27804094-6280-4f24-a5c7-0a0dc5ff6561\") " pod="openshift-ingress-canary/ingress-canary-mwkpt"
Jul 31 14:48:58.130590 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.130503    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-tmp-dir\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.130590 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.130546    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27804094-6280-4f24-a5c7-0a0dc5ff6561-cert\") pod \"ingress-canary-mwkpt\" (UID: \"27804094-6280-4f24-a5c7-0a0dc5ff6561\") " pod="openshift-ingress-canary/ingress-canary-mwkpt"
Jul 31 14:48:58.130590 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:58.130545    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:58.130590 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.130574    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-config-volume\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.130972 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:58.130621    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls podName:51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:58.630600002 +0000 UTC m=+35.150024041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls") pod "dns-default-hcnkz" (UID: "51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:58.131042 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.131024    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-tmp-dir\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.131319 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.131282    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-config-volume\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.135331 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.135312    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/27804094-6280-4f24-a5c7-0a0dc5ff6561-cert\") pod \"ingress-canary-mwkpt\" (UID: \"27804094-6280-4f24-a5c7-0a0dc5ff6561\") " pod="openshift-ingress-canary/ingress-canary-mwkpt"
Jul 31 14:48:58.140174 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.140137    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjkf4\" (UniqueName: \"kubernetes.io/projected/27804094-6280-4f24-a5c7-0a0dc5ff6561-kube-api-access-mjkf4\") pod \"ingress-canary-mwkpt\" (UID: \"27804094-6280-4f24-a5c7-0a0dc5ff6561\") " pod="openshift-ingress-canary/ingress-canary-mwkpt"
Jul 31 14:48:58.140281 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.140230    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slvf2\" (UniqueName: \"kubernetes.io/projected/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-kube-api-access-slvf2\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.238569 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.238535    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-mwkpt"
Jul 31 14:48:58.635025 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.634985    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:58.635234 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:58.635141    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:58.635303 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:58.635235    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls podName:51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5 nodeName:}" failed. No retries permitted until 2026-07-31 14:48:59.635213187 +0000 UTC m=+36.154637211 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls") pod "dns-default-hcnkz" (UID: "51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5") : secret "dns-default-metrics-tls" not found
Jul 31 14:48:58.955862 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.955779    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:48:58.958354 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.958328    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:48:58.958504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:58.958459    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jrzg7\""
Jul 31 14:48:59.398329 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:59.398133    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-mwkpt"]
Jul 31 14:48:59.459460 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:48:59.459418    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27804094_6280_4f24_a5c7_0a0dc5ff6561.slice/crio-b68da6d8f7f0779995b67da5e55034d581635a1f106b893826dbb64a245ae0b0 WatchSource:0}: Error finding container b68da6d8f7f0779995b67da5e55034d581635a1f106b893826dbb64a245ae0b0: Status 404 returned error can't find the container with id b68da6d8f7f0779995b67da5e55034d581635a1f106b893826dbb64a245ae0b0
Jul 31 14:48:59.642465 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:48:59.642418    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:48:59.642637 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:59.642603    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:48:59.642707 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:48:59.642689    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls podName:51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:01.64266633 +0000 UTC m=+38.162090347 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls") pod "dns-default-hcnkz" (UID: "51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:00.261325 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:00.261285    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf493cbb-ad21-4177-b194-d73d297a4cb0" containerID="e9747fbaa5c90a68c65e2aeeb9653324f88af285a779554733b676c3d2914e72" exitCode=0
Jul 31 14:49:00.261502 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:00.261323    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerDied","Data":"e9747fbaa5c90a68c65e2aeeb9653324f88af285a779554733b676c3d2914e72"}
Jul 31 14:49:00.262703 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:00.262610    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mwkpt" event={"ID":"27804094-6280-4f24-a5c7-0a0dc5ff6561","Type":"ContainerStarted","Data":"b68da6d8f7f0779995b67da5e55034d581635a1f106b893826dbb64a245ae0b0"}
Jul 31 14:49:01.268916 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:01.268883    2561 generic.go:356] "Generic (PLEG): container finished" podID="bf493cbb-ad21-4177-b194-d73d297a4cb0" containerID="6ff400c69d5b80ab3b5c5035c6db0b099a70417b6cdafc0009a5da683928e125" exitCode=0
Jul 31 14:49:01.269391 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:01.268935    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerDied","Data":"6ff400c69d5b80ab3b5c5035c6db0b099a70417b6cdafc0009a5da683928e125"}
Jul 31 14:49:01.659263 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:01.659229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:49:01.659426 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:01.659405    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:01.659512 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:01.659499    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls podName:51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:05.659476577 +0000 UTC m=+42.178900599 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls") pod "dns-default-hcnkz" (UID: "51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:02.274234 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:02.273917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-mwkpt" event={"ID":"27804094-6280-4f24-a5c7-0a0dc5ff6561","Type":"ContainerStarted","Data":"d860f9431b12624e859c6773d59be792926243eaa3c1098e7d5f6fa943b9bb3b"}
Jul 31 14:49:02.278226 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:02.278187    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" event={"ID":"bf493cbb-ad21-4177-b194-d73d297a4cb0","Type":"ContainerStarted","Data":"266d07e51a228144b10095b28ab8be7d6ae3d6626796586ebe795c3e23259000"}
Jul 31 14:49:02.298280 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:02.298215    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-mwkpt" podStartSLOduration=3.093807565 podStartE2EDuration="5.298189886s" totalImagesPullingTime="2.204382321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:02.298085546 +0000 UTC m=+38.817509584" watchObservedRunningTime="2026-07-31 14:49:02.298189886 +0000 UTC m=+38.817613924"
Jul 31 14:49:02.351587 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:02.351526    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-7ctdq" podStartSLOduration=8.363177421 podStartE2EDuration="38.351507473s" totalImagesPullingTime="29.988330052s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:02.348579061 +0000 UTC m=+38.868003101" watchObservedRunningTime="2026-07-31 14:49:02.351507473 +0000 UTC m=+38.870931511"
Jul 31 14:49:05.688106 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:05.688061    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:49:05.688566 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:05.688192    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:05.688566 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:05.688244    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls podName:51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:13.688230637 +0000 UTC m=+50.207654653 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls") pod "dns-default-hcnkz" (UID: "51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:13.744846 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.744795    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:49:13.745322 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:13.744960    2561 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 31 14:49:13.745322 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:13.745040    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls podName:51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.74502376 +0000 UTC m=+66.264447776 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls") pod "dns-default-hcnkz" (UID: "51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5") : secret "dns-default-metrics-tls" not found
Jul 31 14:49:13.781022 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.780987    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-77696685b-cx286"]
Jul 31 14:49:13.793631 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.793608    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:13.794862 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.794840    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-cx286"]
Jul 31 14:49:13.796304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.796283    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 31 14:49:13.797028 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.797010    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6dlsx\""
Jul 31 14:49:13.797131 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.797040    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 31 14:49:13.797131 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.797011    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 31 14:49:13.797263 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.797204    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 31 14:49:13.805515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.805495    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 31 14:49:13.904717 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.904578    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l"]
Jul 31 14:49:13.917924 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.917896    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l"
Jul 31 14:49:13.921499 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.921476    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-hkrlh\""
Jul 31 14:49:13.926769 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.926749    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l"]
Jul 31 14:49:13.945877 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.945855    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb00f3ae-bc95-452c-b977-0a857611efcd-serving-cert\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:13.945877 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.945882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c75m\" (UniqueName: \"kubernetes.io/projected/bb00f3ae-bc95-452c-b977-0a857611efcd-kube-api-access-9c75m\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:13.946067 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.945915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb00f3ae-bc95-452c-b977-0a857611efcd-trusted-ca-bundle\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:13.946067 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.946023    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb00f3ae-bc95-452c-b977-0a857611efcd-tmp\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:13.946177 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.946076    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb00f3ae-bc95-452c-b977-0a857611efcd-snapshots\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:13.946177 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:13.946117    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb00f3ae-bc95-452c-b977-0a857611efcd-service-ca-bundle\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.011235 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.011149    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-d892t"]
Jul 31 14:49:14.023452 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.023431    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.026680 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.026659    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 31 14:49:14.030733 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.030717    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 31 14:49:14.031061 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.031047    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8grv9\""
Jul 31 14:49:14.031450 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.031436    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 31 14:49:14.036077 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.036058    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 31 14:49:14.040026 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.039999    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-d892t"]
Jul 31 14:49:14.046737 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.046717    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb00f3ae-bc95-452c-b977-0a857611efcd-trusted-ca-bundle\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.046831 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.046747    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whmz8\" (UniqueName: \"kubernetes.io/projected/f3a91903-3e85-450e-a8af-670aaddd059b-kube-api-access-whmz8\") pod \"network-check-source-65bdbbff45-88r5l\" (UID: \"f3a91903-3e85-450e-a8af-670aaddd059b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l"
Jul 31 14:49:14.046831 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.046819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb00f3ae-bc95-452c-b977-0a857611efcd-tmp\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.046970 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.046957    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb00f3ae-bc95-452c-b977-0a857611efcd-snapshots\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.047005 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.046992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb00f3ae-bc95-452c-b977-0a857611efcd-service-ca-bundle\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.047038 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.047023    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb00f3ae-bc95-452c-b977-0a857611efcd-serving-cert\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.047072 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.047044    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c75m\" (UniqueName: \"kubernetes.io/projected/bb00f3ae-bc95-452c-b977-0a857611efcd-kube-api-access-9c75m\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.047283 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.047263    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bb00f3ae-bc95-452c-b977-0a857611efcd-tmp\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.047595 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.047576    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/bb00f3ae-bc95-452c-b977-0a857611efcd-snapshots\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.047687 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.047621    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb00f3ae-bc95-452c-b977-0a857611efcd-trusted-ca-bundle\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.047687 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.047631    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/bb00f3ae-bc95-452c-b977-0a857611efcd-service-ca-bundle\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.050867 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.050848    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 31 14:49:14.051060 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.051043    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bb00f3ae-bc95-452c-b977-0a857611efcd-serving-cert\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.060558 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.060534    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"]
Jul 31 14:49:14.074907 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.074884    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.090006 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.089979    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c75m\" (UniqueName: \"kubernetes.io/projected/bb00f3ae-bc95-452c-b977-0a857611efcd-kube-api-access-9c75m\") pod \"insights-operator-77696685b-cx286\" (UID: \"bb00f3ae-bc95-452c-b977-0a857611efcd\") " pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.090883 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.090860    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 31 14:49:14.090963 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.090862    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 31 14:49:14.099136 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.099112    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c8rc5\""
Jul 31 14:49:14.101891 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.101873    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 31 14:49:14.102412 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.102397    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 31 14:49:14.105396 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.105381    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-77696685b-cx286"
Jul 31 14:49:14.113006 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.112987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"]
Jul 31 14:49:14.148028 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.148004    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/865fb521-3311-48c8-9f85-3b722f1ad540-trusted-ca\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.148149 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.148038    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rq7kg\" (UniqueName: \"kubernetes.io/projected/865fb521-3311-48c8-9f85-3b722f1ad540-kube-api-access-rq7kg\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.148149 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.148060    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/865fb521-3311-48c8-9f85-3b722f1ad540-config\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.148238 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.148199    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/865fb521-3311-48c8-9f85-3b722f1ad540-serving-cert\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.148238 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.148230    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whmz8\" (UniqueName: \"kubernetes.io/projected/f3a91903-3e85-450e-a8af-670aaddd059b-kube-api-access-whmz8\") pod \"network-check-source-65bdbbff45-88r5l\" (UID: \"f3a91903-3e85-450e-a8af-670aaddd059b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l"
Jul 31 14:49:14.180482 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.180429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whmz8\" (UniqueName: \"kubernetes.io/projected/f3a91903-3e85-450e-a8af-670aaddd059b-kube-api-access-whmz8\") pod \"network-check-source-65bdbbff45-88r5l\" (UID: \"f3a91903-3e85-450e-a8af-670aaddd059b\") " pod="openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l"
Jul 31 14:49:14.227443 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.227415    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l"
Jul 31 14:49:14.248750 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.248724    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/865fb521-3311-48c8-9f85-3b722f1ad540-serving-cert\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.248880 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.248801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-image-registry-private-configuration\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.248880 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.248822    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.248880 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.248847    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-registry-certificates\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.248880 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.248869    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/865fb521-3311-48c8-9f85-3b722f1ad540-trusted-ca\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.249093 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.248896    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rq7kg\" (UniqueName: \"kubernetes.io/projected/865fb521-3311-48c8-9f85-3b722f1ad540-kube-api-access-rq7kg\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.249093 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.248941    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-installation-pull-secrets\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.249093 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.249065    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-bound-sa-token\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.249284 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.249102    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/865fb521-3311-48c8-9f85-3b722f1ad540-config\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.249284 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.249240    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cb1afc0-048c-4408-bf48-974425ee4843-ca-trust-extracted\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.249284 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.249275    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-trusted-ca\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.249431 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.249301    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgqqd\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-kube-api-access-fgqqd\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.250019 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.249983    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/865fb521-3311-48c8-9f85-3b722f1ad540-trusted-ca\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.250101 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.250027    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/865fb521-3311-48c8-9f85-3b722f1ad540-config\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.251401 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.251380    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/865fb521-3311-48c8-9f85-3b722f1ad540-serving-cert\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.257041 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.257021    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-77696685b-cx286"]
Jul 31 14:49:14.262941 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:14.262889    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb00f3ae_bc95_452c_b977_0a857611efcd.slice/crio-cda2fc502ed041e531a33329a142e244fbda7a6c0d32b5ad5a4b4a39476b9e81 WatchSource:0}: Error finding container cda2fc502ed041e531a33329a142e244fbda7a6c0d32b5ad5a4b4a39476b9e81: Status 404 returned error can't find the container with id cda2fc502ed041e531a33329a142e244fbda7a6c0d32b5ad5a4b4a39476b9e81
Jul 31 14:49:14.267304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.267285    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rq7kg\" (UniqueName: \"kubernetes.io/projected/865fb521-3311-48c8-9f85-3b722f1ad540-kube-api-access-rq7kg\") pod \"console-operator-5ccd6b445f-d892t\" (UID: \"865fb521-3311-48c8-9f85-3b722f1ad540\") " pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.308822 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.308787    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-cx286" event={"ID":"bb00f3ae-bc95-452c-b977-0a857611efcd","Type":"ContainerStarted","Data":"cda2fc502ed041e531a33329a142e244fbda7a6c0d32b5ad5a4b4a39476b9e81"}
Jul 31 14:49:14.332081 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.332004    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:14.350424 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350395    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-bound-sa-token\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.350698 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cb1afc0-048c-4408-bf48-974425ee4843-ca-trust-extracted\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.350698 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350478    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-trusted-ca\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.350698 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350495    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgqqd\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-kube-api-access-fgqqd\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.350698 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350531    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-image-registry-private-configuration\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.350698 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350577    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.350698 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350609    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-registry-certificates\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.350698 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350651    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-installation-pull-secrets\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.351030 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:14.350723    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:49:14.351030 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:14.350750    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv: secret "image-registry-tls" not found
Jul 31 14:49:14.351030 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:14.350823    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls podName:8cb1afc0-048c-4408-bf48-974425ee4843 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:14.850797188 +0000 UTC m=+51.370221221 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls") pod "image-registry-5b7f7fbdcd-tpjfv" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843") : secret "image-registry-tls" not found
Jul 31 14:49:14.351030 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.350911    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cb1afc0-048c-4408-bf48-974425ee4843-ca-trust-extracted\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.351385 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.351361    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-registry-certificates\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.351573 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.351554    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-trusted-ca\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.353442 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.353424    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-installation-pull-secrets\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.353526 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.353453    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-image-registry-private-configuration\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.364749 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.364731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgqqd\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-kube-api-access-fgqqd\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.366783 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.366761    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l"]
Jul 31 14:49:14.369379 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:14.369354    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3a91903_3e85_450e_a8af_670aaddd059b.slice/crio-5b8e3bb4dfc52b3c534bcd06b9c7a3cbac9ccb7139f97ff307338e215f78879d WatchSource:0}: Error finding container 5b8e3bb4dfc52b3c534bcd06b9c7a3cbac9ccb7139f97ff307338e215f78879d: Status 404 returned error can't find the container with id 5b8e3bb4dfc52b3c534bcd06b9c7a3cbac9ccb7139f97ff307338e215f78879d
Jul 31 14:49:14.370112 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.370092    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-bound-sa-token\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.483776 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.483743    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-5ccd6b445f-d892t"]
Jul 31 14:49:14.491272 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:14.491247    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod865fb521_3311_48c8_9f85_3b722f1ad540.slice/crio-dc0f820f12f82e8c8974e6793edf23f0b067ef6ff1557d824ed4d540e088189f WatchSource:0}: Error finding container dc0f820f12f82e8c8974e6793edf23f0b067ef6ff1557d824ed4d540e088189f: Status 404 returned error can't find the container with id dc0f820f12f82e8c8974e6793edf23f0b067ef6ff1557d824ed4d540e088189f
Jul 31 14:49:14.854366 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:14.854321    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:14.854819 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:14.854516    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:49:14.854819 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:14.854539    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv: secret "image-registry-tls" not found
Jul 31 14:49:14.854819 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:14.854606    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls podName:8cb1afc0-048c-4408-bf48-974425ee4843 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:15.854590443 +0000 UTC m=+52.374014478 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls") pod "image-registry-5b7f7fbdcd-tpjfv" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843") : secret "image-registry-tls" not found
Jul 31 14:49:15.314304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:15.314221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t" event={"ID":"865fb521-3311-48c8-9f85-3b722f1ad540","Type":"ContainerStarted","Data":"dc0f820f12f82e8c8974e6793edf23f0b067ef6ff1557d824ed4d540e088189f"}
Jul 31 14:49:15.316418 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:15.316354    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l" event={"ID":"f3a91903-3e85-450e-a8af-670aaddd059b","Type":"ContainerStarted","Data":"5b8e3bb4dfc52b3c534bcd06b9c7a3cbac9ccb7139f97ff307338e215f78879d"}
Jul 31 14:49:15.864075 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:15.864031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:15.864543 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:15.864227    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:49:15.864543 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:15.864246    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv: secret "image-registry-tls" not found
Jul 31 14:49:15.864543 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:15.864304    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls podName:8cb1afc0-048c-4408-bf48-974425ee4843 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:17.864285507 +0000 UTC m=+54.383709526 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls") pod "image-registry-5b7f7fbdcd-tpjfv" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843") : secret "image-registry-tls" not found
Jul 31 14:49:17.880671 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:17.880637    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:17.881107 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:17.880765    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:49:17.881107 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:17.880776    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv: secret "image-registry-tls" not found
Jul 31 14:49:17.881107 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:17.880824    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls podName:8cb1afc0-048c-4408-bf48-974425ee4843 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:21.880810159 +0000 UTC m=+58.400234176 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls") pod "image-registry-5b7f7fbdcd-tpjfv" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843") : secret "image-registry-tls" not found
Jul 31 14:49:19.330138 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.330112    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/0.log"
Jul 31 14:49:19.330587 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.330169    2561 generic.go:356] "Generic (PLEG): container finished" podID="865fb521-3311-48c8-9f85-3b722f1ad540" containerID="699ff81e98cf690054a631dfedb2c05db88b34f94350459dbdc2e855b876c8b7" exitCode=255
Jul 31 14:49:19.330587 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.330244    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t" event={"ID":"865fb521-3311-48c8-9f85-3b722f1ad540","Type":"ContainerDied","Data":"699ff81e98cf690054a631dfedb2c05db88b34f94350459dbdc2e855b876c8b7"}
Jul 31 14:49:19.330587 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.330494    2561 scope.go:117] "RemoveContainer" containerID="699ff81e98cf690054a631dfedb2c05db88b34f94350459dbdc2e855b876c8b7"
Jul 31 14:49:19.331830 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.331801    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l" event={"ID":"f3a91903-3e85-450e-a8af-670aaddd059b","Type":"ContainerStarted","Data":"2eae71a2757f381ea369085f7a036137b021c0565f3939b11b580d1817e363de"}
Jul 31 14:49:19.333406 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.333379    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-cx286" event={"ID":"bb00f3ae-bc95-452c-b977-0a857611efcd","Type":"ContainerStarted","Data":"8aa5f3e425b6a629ebb1d64179fc0807ac4e5c2172f0e8f00999217d1e0cce65"}
Jul 31 14:49:19.375401 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.375349    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-77696685b-cx286" podStartSLOduration=1.811454866 podStartE2EDuration="6.375328755s" totalImagesPullingTime="4.563873889s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:19.374141546 +0000 UTC m=+55.893565588" watchObservedRunningTime="2026-07-31 14:49:19.375328755 +0000 UTC m=+55.894752793"
Jul 31 14:49:19.405585 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.405533    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-65bdbbff45-88r5l" podStartSLOduration=1.9476905100000002 podStartE2EDuration="6.405512081s" totalImagesPullingTime="4.457821571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:19.404624169 +0000 UTC m=+55.924048208" watchObservedRunningTime="2026-07-31 14:49:19.405512081 +0000 UTC m=+55.924936120"
Jul 31 14:49:19.646644 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.646569    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c"]
Jul 31 14:49:19.663484 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.663450    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c"
Jul 31 14:49:19.667612 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.667585    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8dq5v\""
Jul 31 14:49:19.668321 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.668298    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 31 14:49:19.668464 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.668445    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c"]
Jul 31 14:49:19.668629 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.668611    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 31 14:49:19.796033 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.796000    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vncfv\" (UniqueName: \"kubernetes.io/projected/d97d6c88-4d9b-4ca8-bc7c-87e814b576cf-kube-api-access-vncfv\") pod \"migrator-68bc745d87-mrt5c\" (UID: \"d97d6c88-4d9b-4ca8-bc7c-87e814b576cf\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c"
Jul 31 14:49:19.896773 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.896691    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vncfv\" (UniqueName: \"kubernetes.io/projected/d97d6c88-4d9b-4ca8-bc7c-87e814b576cf-kube-api-access-vncfv\") pod \"migrator-68bc745d87-mrt5c\" (UID: \"d97d6c88-4d9b-4ca8-bc7c-87e814b576cf\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c"
Jul 31 14:49:19.920512 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.920486    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vncfv\" (UniqueName: \"kubernetes.io/projected/d97d6c88-4d9b-4ca8-bc7c-87e814b576cf-kube-api-access-vncfv\") pod \"migrator-68bc745d87-mrt5c\" (UID: \"d97d6c88-4d9b-4ca8-bc7c-87e814b576cf\") " pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c"
Jul 31 14:49:19.973718 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:19.973683    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c"
Jul 31 14:49:20.058430 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.058395    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4b8hs"]
Jul 31 14:49:20.083145 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.082495    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.084579 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.084556    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4b8hs"]
Jul 31 14:49:20.085212 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.085194    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 31 14:49:20.085938 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.085920    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 31 14:49:20.086455 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.086044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j66bv\""
Jul 31 14:49:20.121944 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.121919    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c"]
Jul 31 14:49:20.124707 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:20.124678    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd97d6c88_4d9b_4ca8_bc7c_87e814b576cf.slice/crio-a5542aad4f22649fce91a5ac91d182e8af592d4d33916c9774cf9092b7d8ba77 WatchSource:0}: Error finding container a5542aad4f22649fce91a5ac91d182e8af592d4d33916c9774cf9092b7d8ba77: Status 404 returned error can't find the container with id a5542aad4f22649fce91a5ac91d182e8af592d4d33916c9774cf9092b7d8ba77
Jul 31 14:49:20.199093 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.199028    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.199093 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.199082    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-data-volume\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.199281 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.199184    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-crio-socket\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.199317 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.199265    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.199352 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.199334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7m6d5\" (UniqueName: \"kubernetes.io/projected/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-api-access-7m6d5\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.300145 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.300112    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.300315 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.300195    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7m6d5\" (UniqueName: \"kubernetes.io/projected/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-api-access-7m6d5\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.300315 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.300229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.300315 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.300257    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-data-volume\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.300315 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.300266    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:20.300503 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.300328    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:20.300503 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.300342    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:20.800321882 +0000 UTC m=+57.319745900 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:20.300503 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.300279    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-crio-socket\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.300503 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.300391    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:20.800376701 +0000 UTC m=+57.319800720 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:20.300503 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.300426    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-crio-socket\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.300661 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.300561    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-data-volume\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.319782 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.319759    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7m6d5\" (UniqueName: \"kubernetes.io/projected/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-api-access-7m6d5\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.338000 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.337975    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 14:49:20.338493 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.338479    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/0.log"
Jul 31 14:49:20.338549 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.338517    2561 generic.go:356] "Generic (PLEG): container finished" podID="865fb521-3311-48c8-9f85-3b722f1ad540" containerID="51ac33d253f761a29100cc67fce08c5a822df2819ea714f3515ce785350aed94" exitCode=255
Jul 31 14:49:20.338634 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.338609    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t" event={"ID":"865fb521-3311-48c8-9f85-3b722f1ad540","Type":"ContainerDied","Data":"51ac33d253f761a29100cc67fce08c5a822df2819ea714f3515ce785350aed94"}
Jul 31 14:49:20.338703 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.338688    2561 scope.go:117] "RemoveContainer" containerID="699ff81e98cf690054a631dfedb2c05db88b34f94350459dbdc2e855b876c8b7"
Jul 31 14:49:20.338907 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.338890    2561 scope.go:117] "RemoveContainer" containerID="51ac33d253f761a29100cc67fce08c5a822df2819ea714f3515ce785350aed94"
Jul 31 14:49:20.339148 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.339098    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-d892t_openshift-console-operator(865fb521-3311-48c8-9f85-3b722f1ad540)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t" podUID="865fb521-3311-48c8-9f85-3b722f1ad540"
Jul 31 14:49:20.341297 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.340103    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c" event={"ID":"d97d6c88-4d9b-4ca8-bc7c-87e814b576cf","Type":"ContainerStarted","Data":"a5542aad4f22649fce91a5ac91d182e8af592d4d33916c9774cf9092b7d8ba77"}
Jul 31 14:49:20.805229 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.805191    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.805424 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.805300    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:20.805424 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.805357    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:20.805424 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.805417    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:20.805577 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.805432    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:21.805412952 +0000 UTC m=+58.324836983 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:20.805577 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:20.805472    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:21.805455399 +0000 UTC m=+58.324879414 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:20.973761 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.973724    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-4wxdc"]
Jul 31 14:49:20.992402 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.992375    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:20.995237 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.995137    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-ktf9r\""
Jul 31 14:49:20.995363 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.995261    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 31 14:49:20.995363 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.995282    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-4wxdc"]
Jul 31 14:49:20.995984 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.995968    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 31 14:49:20.996078 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.996016    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 31 14:49:20.996128 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:20.996022    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 31 14:49:21.107372 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.107297    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-signing-cabundle\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.107372 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.107335    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sdd5\" (UniqueName: \"kubernetes.io/projected/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-kube-api-access-7sdd5\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.107567 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.107476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-signing-key\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.208886 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.208831    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-signing-cabundle\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.208886 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.208886    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sdd5\" (UniqueName: \"kubernetes.io/projected/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-kube-api-access-7sdd5\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.209139 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.208972    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-signing-key\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.209694 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.209659    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-signing-cabundle\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.211993 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.211969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-signing-key\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.218090 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.218059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sdd5\" (UniqueName: \"kubernetes.io/projected/b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8-kube-api-access-7sdd5\") pod \"service-ca-98c88fbb5-4wxdc\" (UID: \"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8\") " pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.303550 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.303514    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc"
Jul 31 14:49:21.343400 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.343379    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 14:49:21.343747 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.343732    2561 scope.go:117] "RemoveContainer" containerID="51ac33d253f761a29100cc67fce08c5a822df2819ea714f3515ce785350aed94"
Jul 31 14:49:21.343908 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:21.343890    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-d892t_openshift-console-operator(865fb521-3311-48c8-9f85-3b722f1ad540)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t" podUID="865fb521-3311-48c8-9f85-3b722f1ad540"
Jul 31 14:49:21.498984 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.498952    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-98c88fbb5-4wxdc"]
Jul 31 14:49:21.501864 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:21.501840    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5817a3d_4d83_4b71_b3f8_6c3acd70ffd8.slice/crio-4073e5633c74c67eef7bbefdb6f6b6857fbd3a8626002eb8bfc3f6471e88663d WatchSource:0}: Error finding container 4073e5633c74c67eef7bbefdb6f6b6857fbd3a8626002eb8bfc3f6471e88663d: Status 404 returned error can't find the container with id 4073e5633c74c67eef7bbefdb6f6b6857fbd3a8626002eb8bfc3f6471e88663d
Jul 31 14:49:21.814928 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.814903    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:21.815026 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.814992    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:21.815086 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:21.815041    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:21.815122 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:21.815098    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:23.815083444 +0000 UTC m=+60.334507465 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:21.815122 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:21.815103    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:21.815219 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:21.815168    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:23.815139632 +0000 UTC m=+60.334563649 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:21.915858 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:21.915828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:21.915995 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:21.915974    2561 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 31 14:49:21.916050 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:21.915997    2561 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv: secret "image-registry-tls" not found
Jul 31 14:49:21.916092 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:21.916051    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls podName:8cb1afc0-048c-4408-bf48-974425ee4843 nodeName:}" failed. No retries permitted until 2026-07-31 14:49:29.91603374 +0000 UTC m=+66.435457757 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls") pod "image-registry-5b7f7fbdcd-tpjfv" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843") : secret "image-registry-tls" not found
Jul 31 14:49:22.350270 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:22.350226    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c" event={"ID":"d97d6c88-4d9b-4ca8-bc7c-87e814b576cf","Type":"ContainerStarted","Data":"86bc35c23700aff2404ea4258400caf25ef8f645e92dd443af34dcf32d9512b5"}
Jul 31 14:49:22.350756 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:22.350301    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c" event={"ID":"d97d6c88-4d9b-4ca8-bc7c-87e814b576cf","Type":"ContainerStarted","Data":"2160ec9cb7f810c1a931155cdc9ba67610da083af6dc3f946c8ba8ecc47f1aa8"}
Jul 31 14:49:22.351537 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:22.351507    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc" event={"ID":"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8","Type":"ContainerStarted","Data":"4073e5633c74c67eef7bbefdb6f6b6857fbd3a8626002eb8bfc3f6471e88663d"}
Jul 31 14:49:22.368526 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:22.368475    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-68bc745d87-mrt5c" podStartSLOduration=1.776189181 podStartE2EDuration="3.368457476s" totalImagesPullingTime="1.592268295s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:22.367229183 +0000 UTC m=+58.886653225" watchObservedRunningTime="2026-07-31 14:49:22.368457476 +0000 UTC m=+58.887881514"
Jul 31 14:49:22.872055 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:22.872022    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wr8xz_98896ea7-ed6d-451c-a61b-e74ab2b1cdf9/dns-node-resolver/0.log"
Jul 31 14:49:23.357058 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:23.357019    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc" event={"ID":"b5817a3d-4d83-4b71-b3f8-6c3acd70ffd8","Type":"ContainerStarted","Data":"796ff55f9d3cf7bda62193668be7db786991fbe8419de49823bcbc14a096f439"}
Jul 31 14:49:23.377471 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:23.377379    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-98c88fbb5-4wxdc" podStartSLOduration=1.765740626 podStartE2EDuration="3.37735789s" totalImagesPullingTime="1.611617264s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:23.376580553 +0000 UTC m=+59.896004591" watchObservedRunningTime="2026-07-31 14:49:23.37735789 +0000 UTC m=+59.896781931"
Jul 31 14:49:23.672509 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:23.672434    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lz89t_68e2f225-4cc2-465b-b01b-6f1e8142b94b/node-ca/0.log"
Jul 31 14:49:23.832132 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:23.832096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:23.832322 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:23.832217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:23.832322 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:23.832232    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:23.832432 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:23.832323    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:27.832305354 +0000 UTC m=+64.351729385 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:23.832432 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:23.832349    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:23.832432 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:23.832432    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:27.832418379 +0000 UTC m=+64.351842412 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:24.333163 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:24.333108    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:24.333337 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:24.333187    2561 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:24.333616 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:24.333601    2561 scope.go:117] "RemoveContainer" containerID="51ac33d253f761a29100cc67fce08c5a822df2819ea714f3515ce785350aed94"
Jul 31 14:49:24.333812 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:24.333786    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-5ccd6b445f-d892t_openshift-console-operator(865fb521-3311-48c8-9f85-3b722f1ad540)\"" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t" podUID="865fb521-3311-48c8-9f85-3b722f1ad540"
Jul 31 14:49:24.473399 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:24.473361    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mwkpt_27804094-6280-4f24-a5c7-0a0dc5ff6561/serve-healthcheck-canary/0.log"
Jul 31 14:49:24.872676 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:24.872647    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-mrt5c_d97d6c88-4d9b-4ca8-bc7c-87e814b576cf/migrator/0.log"
Jul 31 14:49:25.072082 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:25.072056    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-mrt5c_d97d6c88-4d9b-4ca8-bc7c-87e814b576cf/graceful-termination/0.log"
Jul 31 14:49:26.258350 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:26.258323    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k2dtc"
Jul 31 14:49:27.866670 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:27.866639    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:27.867039 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:27.866704    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:27.867039 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:27.866793    2561 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 31 14:49:27.867039 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:27.866808    2561 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:27.867039 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:27.866862    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:35.866846705 +0000 UTC m=+72.386270724 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : configmap "kube-rbac-proxy" not found
Jul 31 14:49:27.867039 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:27.866878    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls podName:ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a nodeName:}" failed. No retries permitted until 2026-07-31 14:49:35.866871415 +0000 UTC m=+72.386295434 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4b8hs" (UID: "ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a") : secret "insights-runtime-extractor-tls" not found
Jul 31 14:49:29.781760 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.781726    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:49:29.781760 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.781764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:49:29.782265 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.781798    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:49:29.784104 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.784082    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5-metrics-tls\") pod \"dns-default-hcnkz\" (UID: \"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5\") " pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:49:29.784556 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.784537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 31 14:49:29.784556 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.784550    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 31 14:49:29.792797 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:29.792779    2561 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 31 14:49:29.792870 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:49:29.792833    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs podName:bcd6660e-bbc4-47a2-94aa-d65b15d96247 nodeName:}" failed. No retries permitted until 2026-07-31 14:50:33.792818546 +0000 UTC m=+130.312242565 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs") pod "network-metrics-daemon-qpj79" (UID: "bcd6660e-bbc4-47a2-94aa-d65b15d96247") : secret "metrics-daemon-secret" not found
Jul 31 14:49:29.794666 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.794643    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/7236d2d7-d084-40e7-9b8d-cd72aa4243b7-original-pull-secret\") pod \"global-pull-secret-syncer-gfczg\" (UID: \"7236d2d7-d084-40e7-9b8d-cd72aa4243b7\") " pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:49:29.882474 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.882446    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:49:29.884713 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.884690    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4q4l\" (UniqueName: \"kubernetes.io/projected/73a4030c-d937-4fde-b65c-e026541f6cbe-kube-api-access-r4q4l\") pod \"network-check-target-ch9qv\" (UID: \"73a4030c-d937-4fde-b65c-e026541f6cbe\") " pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:49:29.983239 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.983205    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:29.985436 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:29.985418    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"image-registry-5b7f7fbdcd-tpjfv\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") " pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:30.028376 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.028350    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-plcv4\""
Jul 31 14:49:30.036527 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.036474    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:49:30.068482 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.068452    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gfczg"
Jul 31 14:49:30.079055 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.079036    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-wtnft\""
Jul 31 14:49:30.086357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.086210    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:49:30.188139 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.187869    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-hcnkz"]
Jul 31 14:49:30.190839 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:30.190811    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51ef06e7_ca6d_44e7_ab06_0fa2d0adfae5.slice/crio-691c7346467683d36c7ab1b96f5653cc0545eb3d10bf17c11dff490cbd0b95f4 WatchSource:0}: Error finding container 691c7346467683d36c7ab1b96f5653cc0545eb3d10bf17c11dff490cbd0b95f4: Status 404 returned error can't find the container with id 691c7346467683d36c7ab1b96f5653cc0545eb3d10bf17c11dff490cbd0b95f4
Jul 31 14:49:30.223586 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.223557    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gfczg"]
Jul 31 14:49:30.227708 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:30.227682    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7236d2d7_d084_40e7_9b8d_cd72aa4243b7.slice/crio-8a2de06bde8aa8cdca67b77d60f1edc544f67bf1f53f459ec6057cf649780ddf WatchSource:0}: Error finding container 8a2de06bde8aa8cdca67b77d60f1edc544f67bf1f53f459ec6057cf649780ddf: Status 404 returned error can't find the container with id 8a2de06bde8aa8cdca67b77d60f1edc544f67bf1f53f459ec6057cf649780ddf
Jul 31 14:49:30.253887 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.253865    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-ch9qv"]
Jul 31 14:49:30.256782 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:30.256757    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73a4030c_d937_4fde_b65c_e026541f6cbe.slice/crio-c740482341e61dda6f91a73bf65400de3af4fa2eaa88ef4ea6eb4c5cce59a716 WatchSource:0}: Error finding container c740482341e61dda6f91a73bf65400de3af4fa2eaa88ef4ea6eb4c5cce59a716: Status 404 returned error can't find the container with id c740482341e61dda6f91a73bf65400de3af4fa2eaa88ef4ea6eb4c5cce59a716
Jul 31 14:49:30.286629 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.286581    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-c8rc5\""
Jul 31 14:49:30.293497 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.293478    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:30.379644 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.379612    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ch9qv" event={"ID":"73a4030c-d937-4fde-b65c-e026541f6cbe","Type":"ContainerStarted","Data":"1b5fbabc4d2addb889f5ca3dc146e23861c4285a62357afe5ff9423c012b17ff"}
Jul 31 14:49:30.379788 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.379671    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-ch9qv" event={"ID":"73a4030c-d937-4fde-b65c-e026541f6cbe","Type":"ContainerStarted","Data":"c740482341e61dda6f91a73bf65400de3af4fa2eaa88ef4ea6eb4c5cce59a716"}
Jul 31 14:49:30.379788 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.379704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:49:30.380905 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.380871    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gfczg" event={"ID":"7236d2d7-d084-40e7-9b8d-cd72aa4243b7","Type":"ContainerStarted","Data":"8a2de06bde8aa8cdca67b77d60f1edc544f67bf1f53f459ec6057cf649780ddf"}
Jul 31 14:49:30.382055 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.382030    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hcnkz" event={"ID":"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5","Type":"ContainerStarted","Data":"691c7346467683d36c7ab1b96f5653cc0545eb3d10bf17c11dff490cbd0b95f4"}
Jul 31 14:49:30.428236 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.428190    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-ch9qv" podStartSLOduration=66.428172636 podStartE2EDuration="1m6.428172636s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:30.411067507 +0000 UTC m=+66.930491546" watchObservedRunningTime="2026-07-31 14:49:30.428172636 +0000 UTC m=+66.947596668"
Jul 31 14:49:30.429138 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:30.429114    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"]
Jul 31 14:49:30.431592 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:30.431569    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cb1afc0_048c_4408_bf48_974425ee4843.slice/crio-b935fa36526c607c060be11cf66ca9604281c115ca97bf4adf54942cd930116b WatchSource:0}: Error finding container b935fa36526c607c060be11cf66ca9604281c115ca97bf4adf54942cd930116b: Status 404 returned error can't find the container with id b935fa36526c607c060be11cf66ca9604281c115ca97bf4adf54942cd930116b
Jul 31 14:49:31.388187 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:31.388135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv" event={"ID":"8cb1afc0-048c-4408-bf48-974425ee4843","Type":"ContainerStarted","Data":"deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d"}
Jul 31 14:49:31.388628 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:31.388217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv" event={"ID":"8cb1afc0-048c-4408-bf48-974425ee4843","Type":"ContainerStarted","Data":"b935fa36526c607c060be11cf66ca9604281c115ca97bf4adf54942cd930116b"}
Jul 31 14:49:32.394865 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:32.394829    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hcnkz" event={"ID":"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5","Type":"ContainerStarted","Data":"4a5fd1246c808e1cdfe8e4da21892f0d34fbd6d774b06de1de3b1e4b0e6051d4"}
Jul 31 14:49:32.395305 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:32.394897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-hcnkz" event={"ID":"51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5","Type":"ContainerStarted","Data":"821600c5609ec24c6414b9d47678f08993747f35b92c876cefa5900ce57b2892"}
Jul 31 14:49:32.395305 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:32.395252    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:49:32.395305 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:32.395298    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:32.418509 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:32.418465    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-hcnkz" podStartSLOduration=33.759109869 podStartE2EDuration="35.418445736s" totalImagesPullingTime="1.659335867s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:32.417453905 +0000 UTC m=+68.936877945" watchObservedRunningTime="2026-07-31 14:49:32.418445736 +0000 UTC m=+68.937869775"
Jul 31 14:49:32.419273 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:32.419237    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv" podStartSLOduration=19.419225991 podStartE2EDuration="19.419225991s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:31.431999796 +0000 UTC m=+67.951423836" watchObservedRunningTime="2026-07-31 14:49:32.419225991 +0000 UTC m=+68.938650029"
Jul 31 14:49:35.406877 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:35.406840    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gfczg" event={"ID":"7236d2d7-d084-40e7-9b8d-cd72aa4243b7","Type":"ContainerStarted","Data":"b9942259a44110e4ccba998968cdd9c0a54097339d832c2f58f5c6a5b2b102a7"}
Jul 31 14:49:35.424677 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:35.424618    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gfczg" podStartSLOduration=66.943504845 podStartE2EDuration="1m11.424598267s" totalImagesPullingTime="4.481093422s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:35.423206307 +0000 UTC m=+71.942630342" watchObservedRunningTime="2026-07-31 14:49:35.424598267 +0000 UTC m=+71.944022306"
Jul 31 14:49:35.938097 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:35.938064    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:35.938365 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:35.938127    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:35.938631 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:35.938609    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:35.940484 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:35.940465    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4b8hs\" (UID: \"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a\") " pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:35.956917 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:35.956893    2561 scope.go:117] "RemoveContainer" containerID="51ac33d253f761a29100cc67fce08c5a822df2819ea714f3515ce785350aed94"
Jul 31 14:49:35.998613 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:35.998584    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-j66bv\""
Jul 31 14:49:36.004814 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.004796    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4b8hs"
Jul 31 14:49:36.145450 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.145414    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4b8hs"]
Jul 31 14:49:36.148499 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:36.148456    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad36a1aa_e3c4_4af6_ad0f_038b2e66ae0a.slice/crio-1804e7b5a0b3d35ae24d7fa14c3a4563e7c127618a0f82908b8a1d947b0ead16 WatchSource:0}: Error finding container 1804e7b5a0b3d35ae24d7fa14c3a4563e7c127618a0f82908b8a1d947b0ead16: Status 404 returned error can't find the container with id 1804e7b5a0b3d35ae24d7fa14c3a4563e7c127618a0f82908b8a1d947b0ead16
Jul 31 14:49:36.411741 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.411706    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b8hs" event={"ID":"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a","Type":"ContainerStarted","Data":"95ccdf828b2ab063343875712229d755b1aa3dc16097a01a7a498fa6cab3f14d"}
Jul 31 14:49:36.412175 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.411768    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b8hs" event={"ID":"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a","Type":"ContainerStarted","Data":"1804e7b5a0b3d35ae24d7fa14c3a4563e7c127618a0f82908b8a1d947b0ead16"}
Jul 31 14:49:36.413476 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.413454    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 14:49:36.413601 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.413570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t" event={"ID":"865fb521-3311-48c8-9f85-3b722f1ad540","Type":"ContainerStarted","Data":"0e9f4b3512185218d56b30397ec421b803ee45de01f50a0d4b7e30cdfbf8d762"}
Jul 31 14:49:36.413854 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.413816    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:36.434038 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.433988    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t" podStartSLOduration=19.096041174 podStartE2EDuration="23.433972148s" totalImagesPullingTime="4.337930974s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:36.432566895 +0000 UTC m=+72.951990929" watchObservedRunningTime="2026-07-31 14:49:36.433972148 +0000 UTC m=+72.953396186"
Jul 31 14:49:36.611360 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:36.611331    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-5ccd6b445f-d892t"
Jul 31 14:49:37.419608 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:37.419569    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b8hs" event={"ID":"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a","Type":"ContainerStarted","Data":"33aae62300b5c4c1310aba045e5b14047299d4a73fa4f3d943fc21f8c43a5522"}
Jul 31 14:49:38.424548 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:38.424465    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4b8hs" event={"ID":"ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a","Type":"ContainerStarted","Data":"4c2e11c4f6469b8eda2402bcd6d5321c13c27d2d9ed94aa8c16e7124eea82ff1"}
Jul 31 14:49:38.454287 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:38.454239    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4b8hs" podStartSLOduration=16.648422046 podStartE2EDuration="18.45422321s" totalImagesPullingTime="1.805801164s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:38.453411992 +0000 UTC m=+74.972836033" watchObservedRunningTime="2026-07-31 14:49:38.45422321 +0000 UTC m=+74.973647247"
Jul 31 14:49:43.404494 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.404464    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-hcnkz"
Jul 31 14:49:43.905913 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.905886    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-ts756"]
Jul 31 14:49:43.907941 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.907918    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756"
Jul 31 14:49:43.912553 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.912530    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-fqw9w\""
Jul 31 14:49:43.912688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.912556    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 31 14:49:43.913394 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.913370    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 31 14:49:43.938128 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.938108    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-ts756"]
Jul 31 14:49:43.979658 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.979635    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"]
Jul 31 14:49:43.987698 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.987678    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-77c974ffcf-hvbbv"]
Jul 31 14:49:43.989594 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.989578    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-hvbbv"
Jul 31 14:49:43.992047 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.992030    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 31 14:49:43.992115 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.992061    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-xsc4z\""
Jul 31 14:49:43.992115 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.992032    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 31 14:49:43.998948 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.998930    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f353439d-f0e7-4580-86a9-6719172fbcde-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-ts756\" (UID: \"f353439d-f0e7-4580-86a9-6719172fbcde\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756"
Jul 31 14:49:43.999027 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:43.998985    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f353439d-f0e7-4580-86a9-6719172fbcde-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-ts756\" (UID: \"f353439d-f0e7-4580-86a9-6719172fbcde\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756"
Jul 31 14:49:44.005070 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.005051    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-hvbbv"]
Jul 31 14:49:44.092649 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.092624    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-d85694948-dfr68"]
Jul 31 14:49:44.094597 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.094583    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.099413 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.099390    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f353439d-f0e7-4580-86a9-6719172fbcde-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-ts756\" (UID: \"f353439d-f0e7-4580-86a9-6719172fbcde\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756"
Jul 31 14:49:44.099503 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.099436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qd94\" (UniqueName: \"kubernetes.io/projected/3680a77c-3f4d-4032-987f-7cb4ab413ccc-kube-api-access-7qd94\") pod \"downloads-77c974ffcf-hvbbv\" (UID: \"3680a77c-3f4d-4032-987f-7cb4ab413ccc\") " pod="openshift-console/downloads-77c974ffcf-hvbbv"
Jul 31 14:49:44.099553 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.099514    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f353439d-f0e7-4580-86a9-6719172fbcde-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-ts756\" (UID: \"f353439d-f0e7-4580-86a9-6719172fbcde\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756"
Jul 31 14:49:44.100048 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.100025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/f353439d-f0e7-4580-86a9-6719172fbcde-nginx-conf\") pod \"networking-console-plugin-f77f48c6d-ts756\" (UID: \"f353439d-f0e7-4580-86a9-6719172fbcde\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756"
Jul 31 14:49:44.102048 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.102018    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/f353439d-f0e7-4580-86a9-6719172fbcde-networking-console-plugin-cert\") pod \"networking-console-plugin-f77f48c6d-ts756\" (UID: \"f353439d-f0e7-4580-86a9-6719172fbcde\") " pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756"
Jul 31 14:49:44.111984 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.111967    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d85694948-dfr68"]
Jul 31 14:49:44.200559 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d566451-0092-4630-9010-94495d450748-installation-pull-secrets\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.200559 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200527    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qd94\" (UniqueName: \"kubernetes.io/projected/3680a77c-3f4d-4032-987f-7cb4ab413ccc-kube-api-access-7qd94\") pod \"downloads-77c974ffcf-hvbbv\" (UID: \"3680a77c-3f4d-4032-987f-7cb4ab413ccc\") " pod="openshift-console/downloads-77c974ffcf-hvbbv"
Jul 31 14:49:44.200559 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200547    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-registry-tls\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.200762 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200567    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d566451-0092-4630-9010-94495d450748-ca-trust-extracted\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.200762 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200687    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d566451-0092-4630-9010-94495d450748-registry-certificates\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.200762 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200746    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knr2b\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-kube-api-access-knr2b\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.200899 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200768    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d566451-0092-4630-9010-94495d450748-image-registry-private-configuration\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.200899 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200786    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d566451-0092-4630-9010-94495d450748-trusted-ca\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.200899 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.200842    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-bound-sa-token\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.213244 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.213220    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qd94\" (UniqueName: \"kubernetes.io/projected/3680a77c-3f4d-4032-987f-7cb4ab413ccc-kube-api-access-7qd94\") pod \"downloads-77c974ffcf-hvbbv\" (UID: \"3680a77c-3f4d-4032-987f-7cb4ab413ccc\") " pod="openshift-console/downloads-77c974ffcf-hvbbv"
Jul 31 14:49:44.216997 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.216976    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756"
Jul 31 14:49:44.298510 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.298479    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-77c974ffcf-hvbbv"
Jul 31 14:49:44.301695 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.301669    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d566451-0092-4630-9010-94495d450748-installation-pull-secrets\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.301827 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.301714    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-registry-tls\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.301827 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.301750    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d566451-0092-4630-9010-94495d450748-ca-trust-extracted\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.301827 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.301777    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d566451-0092-4630-9010-94495d450748-registry-certificates\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.303028 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.302001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knr2b\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-kube-api-access-knr2b\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.303028 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.302066    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d566451-0092-4630-9010-94495d450748-image-registry-private-configuration\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.303028 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.302099    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d566451-0092-4630-9010-94495d450748-trusted-ca\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.303028 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.302141    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/6d566451-0092-4630-9010-94495d450748-ca-trust-extracted\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.303028 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.302172    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-bound-sa-token\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.303028 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.302781    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/6d566451-0092-4630-9010-94495d450748-registry-certificates\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.304181 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.303959    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6d566451-0092-4630-9010-94495d450748-trusted-ca\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.304648 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.304625    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/6d566451-0092-4630-9010-94495d450748-installation-pull-secrets\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.305076 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.305049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-registry-tls\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.305341 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.305321    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/6d566451-0092-4630-9010-94495d450748-image-registry-private-configuration\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.320227 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.320207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knr2b\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-kube-api-access-knr2b\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.331026 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.331000    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/6d566451-0092-4630-9010-94495d450748-bound-sa-token\") pod \"image-registry-d85694948-dfr68\" (UID: \"6d566451-0092-4630-9010-94495d450748\") " pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.356661 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.354886    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-f77f48c6d-ts756"]
Jul 31 14:49:44.359033 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:44.359006    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf353439d_f0e7_4580_86a9_6719172fbcde.slice/crio-868288028e835151ad484c96664ae62899889b600baba824df265c7fa91f7cc4 WatchSource:0}: Error finding container 868288028e835151ad484c96664ae62899889b600baba824df265c7fa91f7cc4: Status 404 returned error can't find the container with id 868288028e835151ad484c96664ae62899889b600baba824df265c7fa91f7cc4
Jul 31 14:49:44.402947 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.402924    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:44.427804 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.427785    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-77c974ffcf-hvbbv"]
Jul 31 14:49:44.429668 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:44.429646    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3680a77c_3f4d_4032_987f_7cb4ab413ccc.slice/crio-7a5fa7d4a02553d08836b6aef92ba1b6203a357c8d21324651cf994c3f6588be WatchSource:0}: Error finding container 7a5fa7d4a02553d08836b6aef92ba1b6203a357c8d21324651cf994c3f6588be: Status 404 returned error can't find the container with id 7a5fa7d4a02553d08836b6aef92ba1b6203a357c8d21324651cf994c3f6588be
Jul 31 14:49:44.444722 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.444695    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756" event={"ID":"f353439d-f0e7-4580-86a9-6719172fbcde","Type":"ContainerStarted","Data":"868288028e835151ad484c96664ae62899889b600baba824df265c7fa91f7cc4"}
Jul 31 14:49:44.445872 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.445846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-hvbbv" event={"ID":"3680a77c-3f4d-4032-987f-7cb4ab413ccc","Type":"ContainerStarted","Data":"7a5fa7d4a02553d08836b6aef92ba1b6203a357c8d21324651cf994c3f6588be"}
Jul 31 14:49:44.545088 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:44.545067    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-d85694948-dfr68"]
Jul 31 14:49:44.546709 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:49:44.546681    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d566451_0092_4630_9010_94495d450748.slice/crio-bc364db59f4030669182fca163ea676d457c35d4c99e8f2ba14ca731f5c391ae WatchSource:0}: Error finding container bc364db59f4030669182fca163ea676d457c35d4c99e8f2ba14ca731f5c391ae: Status 404 returned error can't find the container with id bc364db59f4030669182fca163ea676d457c35d4c99e8f2ba14ca731f5c391ae
Jul 31 14:49:45.452264 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:45.452126    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d85694948-dfr68" event={"ID":"6d566451-0092-4630-9010-94495d450748","Type":"ContainerStarted","Data":"8b6b11dd90f451550e6dc10014fa0796298b8c04bab2826e4d84110a2776d9cb"}
Jul 31 14:49:45.452264 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:45.452241    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-d85694948-dfr68" event={"ID":"6d566451-0092-4630-9010-94495d450748","Type":"ContainerStarted","Data":"bc364db59f4030669182fca163ea676d457c35d4c99e8f2ba14ca731f5c391ae"}
Jul 31 14:49:45.453003 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:45.452981    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:49:45.479740 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:45.479687    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-d85694948-dfr68" podStartSLOduration=1.4796674 podStartE2EDuration="1.4796674s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:45.478410934 +0000 UTC m=+81.997834984" watchObservedRunningTime="2026-07-31 14:49:45.4796674 +0000 UTC m=+81.999091431"
Jul 31 14:49:46.457783 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:46.457738    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756" event={"ID":"f353439d-f0e7-4580-86a9-6719172fbcde","Type":"ContainerStarted","Data":"e43dcc38ab7228dd3cede0a002612983e03ec8557012fa323ea4ef249877bc59"}
Jul 31 14:49:46.474254 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:46.474142    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-f77f48c6d-ts756" podStartSLOduration=2.17700502 podStartE2EDuration="3.474119969s" totalImagesPullingTime="1.297114949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:49:46.473665669 +0000 UTC m=+82.993089708" watchObservedRunningTime="2026-07-31 14:49:46.474119969 +0000 UTC m=+82.993544007"
Jul 31 14:49:53.985683 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:53.985657    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:49:59.922805 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.922765    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"]
Jul 31 14:49:59.926349 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.926320    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:49:59.930111 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.930046    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 31 14:49:59.930301 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.930182    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 31 14:49:59.930374 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.930357    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 31 14:49:59.930938 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.930796    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 31 14:49:59.931290 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.931226    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 31 14:49:59.932857 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.932727    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-k7x7z\""
Jul 31 14:49:59.933807 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.933788    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 31 14:49:59.953100 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:49:59.953058    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"]
Jul 31 14:50:00.001519 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.000028    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d2bqd"]
Jul 31 14:50:00.002756 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.002734    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.005217 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.005195    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 31 14:50:00.005555 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.005537    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 31 14:50:00.006565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.006315    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-lcpl5\""
Jul 31 14:50:00.006565 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.006366    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 31 14:50:00.047123 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.047088    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6e8069-aaf2-46b5-8e2c-502580f70286-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.047339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.047135    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.047339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.047182    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.047339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.047274    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.047498 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.047334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/da6e8069-aaf2-46b5-8e2c-502580f70286-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.047498 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.047369    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqkq4\" (UniqueName: \"kubernetes.io/projected/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-api-access-dqkq4\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-wtmp\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148599    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148639    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dx5mv\" (UniqueName: \"kubernetes.io/projected/f8a1604f-8087-41ab-b091-44868d7ad176-kube-api-access-dx5mv\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148703    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8a1604f-8087-41ab-b091-44868d7ad176-metrics-client-ca\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148731    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-root\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148787    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6e8069-aaf2-46b5-8e2c-502580f70286-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148817    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148846    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148874    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-sys\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148923    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148966    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-tls\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.148999    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/da6e8069-aaf2-46b5-8e2c-502580f70286-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.149031    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqkq4\" (UniqueName: \"kubernetes.io/projected/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-api-access-dqkq4\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.149062    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-textfile\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.151049 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.149969    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/da6e8069-aaf2-46b5-8e2c-502580f70286-metrics-client-ca\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151996 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:50:00.150460    2561 secret.go:189] Couldn't get secret openshift-monitoring/kube-state-metrics-tls: secret "kube-state-metrics-tls" not found
Jul 31 14:50:00.151996 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:50:00.150543    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-tls podName:da6e8069-aaf2-46b5-8e2c-502580f70286 nodeName:}" failed. No retries permitted until 2026-07-31 14:50:00.65052103 +0000 UTC m=+97.169945047 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-state-metrics-tls" (UniqueName: "kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-tls") pod "kube-state-metrics-6bd9664c4-9mdg5" (UID: "da6e8069-aaf2-46b5-8e2c-502580f70286") : secret "kube-state-metrics-tls" not found
Jul 31 14:50:00.151996 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.151030    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.151996 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.151079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/da6e8069-aaf2-46b5-8e2c-502580f70286-volume-directive-shadow\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.154592 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.154570    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.168302 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.168267    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqkq4\" (UniqueName: \"kubernetes.io/projected/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-api-access-dqkq4\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.249879 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.249663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-tls\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.249879 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.249723    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-textfile\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.249879 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.249762    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-wtmp\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.249879 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.249789    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.249879 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.249821    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.250706 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.250676    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dx5mv\" (UniqueName: \"kubernetes.io/projected/f8a1604f-8087-41ab-b091-44868d7ad176-kube-api-access-dx5mv\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.250831 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.250728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8a1604f-8087-41ab-b091-44868d7ad176-metrics-client-ca\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.250831 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.250764    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-root\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.250831 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.250829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-sys\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.251192 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.251146    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-sys\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.251436 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.251217    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-textfile\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.251570 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.251550    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-wtmp\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.252190 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.252139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-accelerators-collector-config\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.252346 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.252325    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/f8a1604f-8087-41ab-b091-44868d7ad176-root\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.252442 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.252344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f8a1604f-8087-41ab-b091-44868d7ad176-metrics-client-ca\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.254508 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.254480    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-tls\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.261331 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.260928    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/f8a1604f-8087-41ab-b091-44868d7ad176-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.268552 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.268525    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dx5mv\" (UniqueName: \"kubernetes.io/projected/f8a1604f-8087-41ab-b091-44868d7ad176-kube-api-access-dx5mv\") pod \"node-exporter-d2bqd\" (UID: \"f8a1604f-8087-41ab-b091-44868d7ad176\") " pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.316763 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.316719    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d2bqd"
Jul 31 14:50:00.655100 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.655006    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.658261 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.658228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/da6e8069-aaf2-46b5-8e2c-502580f70286-kube-state-metrics-tls\") pod \"kube-state-metrics-6bd9664c4-9mdg5\" (UID: \"da6e8069-aaf2-46b5-8e2c-502580f70286\") " pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.801143 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:50:00.801108    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8a1604f_8087_41ab_b091_44868d7ad176.slice/crio-537783c692c992ce2fb93a677c0b160923bcadab379fc9db2d4c6dbf860b8e00 WatchSource:0}: Error finding container 537783c692c992ce2fb93a677c0b160923bcadab379fc9db2d4c6dbf860b8e00: Status 404 returned error can't find the container with id 537783c692c992ce2fb93a677c0b160923bcadab379fc9db2d4c6dbf860b8e00
Jul 31 14:50:00.838320 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.838292    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"
Jul 31 14:50:00.989335 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:00.989300    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5"]
Jul 31 14:50:00.993412 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:50:00.993378    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda6e8069_aaf2_46b5_8e2c_502580f70286.slice/crio-950f0df4efe27b18db0c7776f98eccc1454ddb2c1d50b90bf07500c42f248008 WatchSource:0}: Error finding container 950f0df4efe27b18db0c7776f98eccc1454ddb2c1d50b90bf07500c42f248008: Status 404 returned error can't find the container with id 950f0df4efe27b18db0c7776f98eccc1454ddb2c1d50b90bf07500c42f248008
Jul 31 14:50:01.391637 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:01.391607    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-ch9qv"
Jul 31 14:50:01.525829 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:01.525796    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-77c974ffcf-hvbbv" event={"ID":"3680a77c-3f4d-4032-987f-7cb4ab413ccc","Type":"ContainerStarted","Data":"3b9dea561ae5924c778aed6003134388dbc6cb82284973f09cca2b4892b81f10"}
Jul 31 14:50:01.526072 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:01.526046    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-77c974ffcf-hvbbv"
Jul 31 14:50:01.528677 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:01.528614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5" event={"ID":"da6e8069-aaf2-46b5-8e2c-502580f70286","Type":"ContainerStarted","Data":"950f0df4efe27b18db0c7776f98eccc1454ddb2c1d50b90bf07500c42f248008"}
Jul 31 14:50:01.530740 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:01.530715    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2bqd" event={"ID":"f8a1604f-8087-41ab-b091-44868d7ad176","Type":"ContainerStarted","Data":"537783c692c992ce2fb93a677c0b160923bcadab379fc9db2d4c6dbf860b8e00"}
Jul 31 14:50:01.538740 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:01.538704    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-77c974ffcf-hvbbv"
Jul 31 14:50:01.546594 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:01.546266    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-77c974ffcf-hvbbv" podStartSLOduration=2.113616191 podStartE2EDuration="18.546248882s" totalImagesPullingTime="16.432632691s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:01.545761636 +0000 UTC m=+98.065185675" watchObservedRunningTime="2026-07-31 14:50:01.546248882 +0000 UTC m=+98.065672921"
Jul 31 14:50:02.538989 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:02.538292    2561 generic.go:356] "Generic (PLEG): container finished" podID="f8a1604f-8087-41ab-b091-44868d7ad176" containerID="d4740296362d76634cadd726a34ce0e830c83b59a85d33cd362d54056e80d4a5" exitCode=0
Jul 31 14:50:02.538989 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:02.538923    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2bqd" event={"ID":"f8a1604f-8087-41ab-b091-44868d7ad176","Type":"ContainerDied","Data":"d4740296362d76634cadd726a34ce0e830c83b59a85d33cd362d54056e80d4a5"}
Jul 31 14:50:02.961770 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:02.961686    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-5fc555f85b-lnp49"]
Jul 31 14:50:02.994023 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:02.993987    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fc555f85b-lnp49"]
Jul 31 14:50:02.994237 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:02.994107    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.002025 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.002000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 31 14:50:03.002438 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.002292    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-qqmmn\""
Jul 31 14:50:03.002640 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.002578    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 31 14:50:03.002890 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.002790    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 31 14:50:03.003015 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.003000    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 31 14:50:03.003297 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.003233    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-41i5uvoro97pd\""
Jul 31 14:50:03.003491 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.003431    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 31 14:50:03.179753 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.179719    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0db1694b-5738-4896-baa6-7385e719200f-metrics-client-ca\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.179864 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.179792    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hml4g\" (UniqueName: \"kubernetes.io/projected/0db1694b-5738-4896-baa6-7385e719200f-kube-api-access-hml4g\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.179864 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.179826    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-tls\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.179977 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.179859    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.179977 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.179915    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.179977 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.179952    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.180124 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.179980    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.180124 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.180049    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-grpc-tls\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.282688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.281458    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-grpc-tls\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.282688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.281522    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0db1694b-5738-4896-baa6-7385e719200f-metrics-client-ca\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.282688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.281592    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hml4g\" (UniqueName: \"kubernetes.io/projected/0db1694b-5738-4896-baa6-7385e719200f-kube-api-access-hml4g\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.282688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.281631    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-tls\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.282688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.281675    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.282688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.281728    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.282688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.281793    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.282688 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.281829    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.283253 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.283226    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/0db1694b-5738-4896-baa6-7385e719200f-metrics-client-ca\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.285825 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.285772    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-grpc-tls\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.286497 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.286340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.287400 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.286591    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.287400 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.286947    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-tls\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.287400 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.287356    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.290314 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.290288    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/0db1694b-5738-4896-baa6-7385e719200f-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.290895 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.290874    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hml4g\" (UniqueName: \"kubernetes.io/projected/0db1694b-5738-4896-baa6-7385e719200f-kube-api-access-hml4g\") pod \"thanos-querier-5fc555f85b-lnp49\" (UID: \"0db1694b-5738-4896-baa6-7385e719200f\") " pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.312006 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.311978    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:03.554058 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.554024    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2bqd" event={"ID":"f8a1604f-8087-41ab-b091-44868d7ad176","Type":"ContainerStarted","Data":"e7b5a7dd8ea61b539f2be761ad68a514eea2378cf2addbf178879d996f6f1a2c"}
Jul 31 14:50:03.554420 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.554115    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d2bqd" event={"ID":"f8a1604f-8087-41ab-b091-44868d7ad176","Type":"ContainerStarted","Data":"032b6f1b7bd4b2f35596f769e031eba438cbfab1ff69e60ae73a6c97a49b8e6e"}
Jul 31 14:50:03.600048 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.599640    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d2bqd" podStartSLOduration=3.430291237 podStartE2EDuration="4.599617372s" totalImagesPullingTime="1.092479536s" totalInitContainerRuntime="76.846599ms" isStatefulPod=true podCreationTimestamp="2026-07-31 14:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:03.597986234 +0000 UTC m=+100.117410271" watchObservedRunningTime="2026-07-31 14:50:03.599617372 +0000 UTC m=+100.119041410"
Jul 31 14:50:03.720788 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:03.720697    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-5fc555f85b-lnp49"]
Jul 31 14:50:03.725179 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:50:03.724682    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db1694b_5738_4896_baa6_7385e719200f.slice/crio-bcbe29004ed471401c072ad99b3d55cebef9d736b37e04fc4531f68e0f3bb8ea WatchSource:0}: Error finding container bcbe29004ed471401c072ad99b3d55cebef9d736b37e04fc4531f68e0f3bb8ea: Status 404 returned error can't find the container with id bcbe29004ed471401c072ad99b3d55cebef9d736b37e04fc4531f68e0f3bb8ea
Jul 31 14:50:04.471969 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.471919    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-57bf66bc-klhxq"]
Jul 31 14:50:04.497074 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.497029    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57bf66bc-klhxq"]
Jul 31 14:50:04.497411 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.497260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.501659 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.501497    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 31 14:50:04.501659 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.501585    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 31 14:50:04.501861 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.501500    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nv8g2\""
Jul 31 14:50:04.501861 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.501805    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-3kaqgcm6u1g0\""
Jul 31 14:50:04.502041 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.502012    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 31 14:50:04.502419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.502267    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 31 14:50:04.562224 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.562100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49" event={"ID":"0db1694b-5738-4896-baa6-7385e719200f","Type":"ContainerStarted","Data":"bcbe29004ed471401c072ad99b3d55cebef9d736b37e04fc4531f68e0f3bb8ea"}
Jul 31 14:50:04.570290 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.570257    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5" event={"ID":"da6e8069-aaf2-46b5-8e2c-502580f70286","Type":"ContainerStarted","Data":"72965c0d861d2d601c49c97404cc41ef9b6f19b60b23662d43fb886ff7ac0ed5"}
Jul 31 14:50:04.570447 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.570319    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5" event={"ID":"da6e8069-aaf2-46b5-8e2c-502580f70286","Type":"ContainerStarted","Data":"46db69762c59063028ede06182003e836fcea5a81e2cdab4f6520c0881bc347b"}
Jul 31 14:50:04.570447 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.570348    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5" event={"ID":"da6e8069-aaf2-46b5-8e2c-502580f70286","Type":"ContainerStarted","Data":"93229975e2e3b509c51addcadc811ec9d17ba3ab1ddf88a4ea29d71abd703ce0"}
Jul 31 14:50:04.594972 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.594923    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"]
Jul 31 14:50:04.595488 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.595459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252aefbd-e10e-4d56-86be-f9fdc401ae16-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.595622 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.595501    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-client-ca-bundle\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.595622 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.595555    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/252aefbd-e10e-4d56-86be-f9fdc401ae16-audit-log\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.595622 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.595602    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-secret-metrics-server-client-certs\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.595792 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.595651    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/252aefbd-e10e-4d56-86be-f9fdc401ae16-metrics-server-audit-profiles\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.595792 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.595692    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-secret-metrics-server-tls\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.595792 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.595722    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnxxr\" (UniqueName: \"kubernetes.io/projected/252aefbd-e10e-4d56-86be-f9fdc401ae16-kube-api-access-gnxxr\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.599428 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.599370    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-6bd9664c4-9mdg5" podStartSLOduration=3.046661793 podStartE2EDuration="5.599356594s" totalImagesPullingTime="2.552694801s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:49:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:04.598347287 +0000 UTC m=+101.117771329" watchObservedRunningTime="2026-07-31 14:50:04.599356594 +0000 UTC m=+101.118780632"
Jul 31 14:50:04.619661 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.619625    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"]
Jul 31 14:50:04.619812 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.619754    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:04.622422 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.622397    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-gcgsh\""
Jul 31 14:50:04.622634 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.622615    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 31 14:50:04.697255 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.697217    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/252aefbd-e10e-4d56-86be-f9fdc401ae16-metrics-server-audit-profiles\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.697435 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.697280    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-secret-metrics-server-tls\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.697435 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.697313    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnxxr\" (UniqueName: \"kubernetes.io/projected/252aefbd-e10e-4d56-86be-f9fdc401ae16-kube-api-access-gnxxr\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.697549 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.697456    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252aefbd-e10e-4d56-86be-f9fdc401ae16-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.697549 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.697487    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-client-ca-bundle\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.697655 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.697597    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/252aefbd-e10e-4d56-86be-f9fdc401ae16-audit-log\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.697706 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.697659    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-secret-metrics-server-client-certs\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.703056 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.702929    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/252aefbd-e10e-4d56-86be-f9fdc401ae16-metrics-server-audit-profiles\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.704306 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.704222    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/252aefbd-e10e-4d56-86be-f9fdc401ae16-audit-log\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.705171 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.704768    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/252aefbd-e10e-4d56-86be-f9fdc401ae16-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.717353 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.717325    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-client-ca-bundle\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.719676 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.719658    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-secret-metrics-server-tls\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.719895 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.719872    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/252aefbd-e10e-4d56-86be-f9fdc401ae16-secret-metrics-server-client-certs\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.720432 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.720402    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnxxr\" (UniqueName: \"kubernetes.io/projected/252aefbd-e10e-4d56-86be-f9fdc401ae16-kube-api-access-gnxxr\") pod \"metrics-server-57bf66bc-klhxq\" (UID: \"252aefbd-e10e-4d56-86be-f9fdc401ae16\") " pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.798336 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.798270    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51073248-1761-40d8-82b9-d54cbac2e7e7-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t6zk8\" (UID: \"51073248-1761-40d8-82b9-d54cbac2e7e7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:04.798533 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.798367    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xvbcq\" (UniqueName: \"kubernetes.io/projected/51073248-1761-40d8-82b9-d54cbac2e7e7-kube-api-access-xvbcq\") pod \"monitoring-plugin-6dffbbd67-t6zk8\" (UID: \"51073248-1761-40d8-82b9-d54cbac2e7e7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:04.813517 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.813136    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:04.899467 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.899430    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51073248-1761-40d8-82b9-d54cbac2e7e7-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t6zk8\" (UID: \"51073248-1761-40d8-82b9-d54cbac2e7e7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:04.899655 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.899590    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xvbcq\" (UniqueName: \"kubernetes.io/projected/51073248-1761-40d8-82b9-d54cbac2e7e7-kube-api-access-xvbcq\") pod \"monitoring-plugin-6dffbbd67-t6zk8\" (UID: \"51073248-1761-40d8-82b9-d54cbac2e7e7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:04.899655 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:50:04.899614    2561 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 31 14:50:04.899767 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:50:04.899698    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/51073248-1761-40d8-82b9-d54cbac2e7e7-monitoring-plugin-cert podName:51073248-1761-40d8-82b9-d54cbac2e7e7 nodeName:}" failed. No retries permitted until 2026-07-31 14:50:05.399674611 +0000 UTC m=+101.919098645 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/51073248-1761-40d8-82b9-d54cbac2e7e7-monitoring-plugin-cert") pod "monitoring-plugin-6dffbbd67-t6zk8" (UID: "51073248-1761-40d8-82b9-d54cbac2e7e7") : secret "monitoring-plugin-cert" not found
Jul 31 14:50:04.912365 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:04.912308    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xvbcq\" (UniqueName: \"kubernetes.io/projected/51073248-1761-40d8-82b9-d54cbac2e7e7-kube-api-access-xvbcq\") pod \"monitoring-plugin-6dffbbd67-t6zk8\" (UID: \"51073248-1761-40d8-82b9-d54cbac2e7e7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:05.034072 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:05.033975    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-57bf66bc-klhxq"]
Jul 31 14:50:05.039559 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:50:05.039526    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod252aefbd_e10e_4d56_86be_f9fdc401ae16.slice/crio-8f0a16d70caa2e966705f802d48663b94e10c163b486bf35cc397d243009fbf3 WatchSource:0}: Error finding container 8f0a16d70caa2e966705f802d48663b94e10c163b486bf35cc397d243009fbf3: Status 404 returned error can't find the container with id 8f0a16d70caa2e966705f802d48663b94e10c163b486bf35cc397d243009fbf3
Jul 31 14:50:05.406523 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:05.406481    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51073248-1761-40d8-82b9-d54cbac2e7e7-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t6zk8\" (UID: \"51073248-1761-40d8-82b9-d54cbac2e7e7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:05.410047 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:05.409999    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/51073248-1761-40d8-82b9-d54cbac2e7e7-monitoring-plugin-cert\") pod \"monitoring-plugin-6dffbbd67-t6zk8\" (UID: \"51073248-1761-40d8-82b9-d54cbac2e7e7\") " pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:05.536852 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:05.536752    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:05.576062 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:05.576022    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq" event={"ID":"252aefbd-e10e-4d56-86be-f9fdc401ae16","Type":"ContainerStarted","Data":"8f0a16d70caa2e966705f802d48663b94e10c163b486bf35cc397d243009fbf3"}
Jul 31 14:50:06.132279 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.132239    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"]
Jul 31 14:50:06.135792 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:50:06.135746    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51073248_1761_40d8_82b9_d54cbac2e7e7.slice/crio-2cde7ee025b0fb281bd9805175c84b1a5b07fde847de504b29624fe7b5f5c48d WatchSource:0}: Error finding container 2cde7ee025b0fb281bd9805175c84b1a5b07fde847de504b29624fe7b5f5c48d: Status 404 returned error can't find the container with id 2cde7ee025b0fb281bd9805175c84b1a5b07fde847de504b29624fe7b5f5c48d
Jul 31 14:50:06.241420 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.241379    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:50:06.261106 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.261063    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.264215 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.264192    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:50:06.265068 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.265045    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 14:50:06.265068 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.265064    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 14:50:06.265267 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.265226    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 14:50:06.266541 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.265438    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4zh25\""
Jul 31 14:50:06.266541 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.265507    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 14:50:06.266541 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.265609    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 14:50:06.266541 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.265722    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 14:50:06.266541 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.265924    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 14:50:06.266541 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.266098    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 14:50:06.267104 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.267075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e47enl3gdkn82\""
Jul 31 14:50:06.267597 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.267425    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 14:50:06.267597 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.267439    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 14:50:06.269241 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.267894    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 14:50:06.269241 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.267968    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 14:50:06.269241 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.269088    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 14:50:06.273100 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.272674    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 14:50:06.314983 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.314947    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-config\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315206 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.314991    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315206 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315018    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315206 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315053    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315206 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj2n6\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-kube-api-access-cj2n6\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315206 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315178    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315227    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315277    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315308    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315334    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315352    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315389    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315426    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315461 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315450    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315840 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315840 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315503    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315840 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315550    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315840 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315609    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315840 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315652    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.315840 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.315681    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418253    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418341    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-config\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418368    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418393    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj2n6\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-kube-api-access-cj2n6\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418746    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418775    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418837    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.418962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.418879    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419611 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419489    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419669 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419721 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419721 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419715    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419822 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419761    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419822 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419788    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419921 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419819    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419921 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419848    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.419921 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419884    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.420101 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419930    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.420101 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.419966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.421422 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.421394    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.423925 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.423290    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.425287 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.425261    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.426126 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.425742    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.426126 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.426059    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.426311 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.426287    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.426577 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.426560    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.433313 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.432542    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.433313 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.433014    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-web-config\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.434702 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.434026    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.434702 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.434448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.434702 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.434664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.437216 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.434914    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-config\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.437216 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.435248    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.437216 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.435347    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.437216 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.435547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-config-out\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.437216 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.435722    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.437216 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.435941    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.444522 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.439798    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj2n6\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-kube-api-access-cj2n6\") pod \"prometheus-k8s-0\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.577042 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.576745    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:06.585440 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.585313    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49" event={"ID":"0db1694b-5738-4896-baa6-7385e719200f","Type":"ContainerStarted","Data":"66b82fb4a131f907821b893d5ca77e61146a7a5083da94bed4cabcd2116a4a91"}
Jul 31 14:50:06.586932 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.586903    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8" event={"ID":"51073248-1761-40d8-82b9-d54cbac2e7e7","Type":"ContainerStarted","Data":"2cde7ee025b0fb281bd9805175c84b1a5b07fde847de504b29624fe7b5f5c48d"}
Jul 31 14:50:06.784302 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:06.784194    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:50:07.469206 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:07.469147    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-d85694948-dfr68"
Jul 31 14:50:07.593321 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:07.593283    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerStarted","Data":"578ff55bbc991aba6db5ef518ad9738d4042799f218c644de9bfe384151f5599"}
Jul 31 14:50:07.595947 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:07.595917    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49" event={"ID":"0db1694b-5738-4896-baa6-7385e719200f","Type":"ContainerStarted","Data":"bb502c272bdb74e2a291c3930d732a9eed36d3e0f9b5338ca7592ba72efd2f48"}
Jul 31 14:50:07.596093 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:07.595992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49" event={"ID":"0db1694b-5738-4896-baa6-7385e719200f","Type":"ContainerStarted","Data":"feb0ff2fbabd484284934633a043581d24203bc6da46f525c260893cec41f3a4"}
Jul 31 14:50:08.602780 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:08.602739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq" event={"ID":"252aefbd-e10e-4d56-86be-f9fdc401ae16","Type":"ContainerStarted","Data":"1f2a361876ead0dcd47c16af620759eaa4f24fd69bd5b76d646410352831aa7e"}
Jul 31 14:50:08.629623 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:08.628439    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq" podStartSLOduration=1.7809035359999998 podStartE2EDuration="4.628412674s" totalImagesPullingTime="2.847509138s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:08.627303027 +0000 UTC m=+105.146727077" watchObservedRunningTime="2026-07-31 14:50:08.628412674 +0000 UTC m=+105.147836712"
Jul 31 14:50:08.998670 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:08.998620    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv" podUID="8cb1afc0-048c-4408-bf48-974425ee4843" containerName="registry" containerID="cri-o://deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d" gracePeriod=30
Jul 31 14:50:10.235058 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.235029    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:50:10.262019 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.261994    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cb1afc0-048c-4408-bf48-974425ee4843-ca-trust-extracted\") pod \"8cb1afc0-048c-4408-bf48-974425ee4843\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") "
Jul 31 14:50:10.262125 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.262038    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-installation-pull-secrets\") pod \"8cb1afc0-048c-4408-bf48-974425ee4843\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") "
Jul 31 14:50:10.262125 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.262076    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fgqqd\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-kube-api-access-fgqqd\") pod \"8cb1afc0-048c-4408-bf48-974425ee4843\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") "
Jul 31 14:50:10.262274 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.262145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-registry-certificates\") pod \"8cb1afc0-048c-4408-bf48-974425ee4843\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") "
Jul 31 14:50:10.262345 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.262324    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-trusted-ca\") pod \"8cb1afc0-048c-4408-bf48-974425ee4843\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") "
Jul 31 14:50:10.262402 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.262358    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-image-registry-private-configuration\") pod \"8cb1afc0-048c-4408-bf48-974425ee4843\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") "
Jul 31 14:50:10.262453 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.262407    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") pod \"8cb1afc0-048c-4408-bf48-974425ee4843\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") "
Jul 31 14:50:10.262453 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.262432    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-bound-sa-token\") pod \"8cb1afc0-048c-4408-bf48-974425ee4843\" (UID: \"8cb1afc0-048c-4408-bf48-974425ee4843\") "
Jul 31 14:50:10.263752 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.263509    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "8cb1afc0-048c-4408-bf48-974425ee4843" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:10.263752 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.263721    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "8cb1afc0-048c-4408-bf48-974425ee4843" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:50:10.264881 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.264852    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "8cb1afc0-048c-4408-bf48-974425ee4843" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:50:10.265403 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.265376    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-kube-api-access-fgqqd" (OuterVolumeSpecName: "kube-api-access-fgqqd") pod "8cb1afc0-048c-4408-bf48-974425ee4843" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843"). InnerVolumeSpecName "kube-api-access-fgqqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:10.265677 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.265650    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "8cb1afc0-048c-4408-bf48-974425ee4843" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:10.266276 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.266241    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "8cb1afc0-048c-4408-bf48-974425ee4843" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:50:10.266625 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.266605    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "8cb1afc0-048c-4408-bf48-974425ee4843" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:50:10.270999 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.270979    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cb1afc0-048c-4408-bf48-974425ee4843-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "8cb1afc0-048c-4408-bf48-974425ee4843" (UID: "8cb1afc0-048c-4408-bf48-974425ee4843"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:50:10.363454 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.363429    2561 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-registry-certificates\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:50:10.363454 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.363454    2561 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8cb1afc0-048c-4408-bf48-974425ee4843-trusted-ca\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:50:10.363604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.363464    2561 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-image-registry-private-configuration\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:50:10.363604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.363472    2561 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-registry-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:50:10.363604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.363482    2561 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-bound-sa-token\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:50:10.363604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.363490    2561 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8cb1afc0-048c-4408-bf48-974425ee4843-ca-trust-extracted\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:50:10.363604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.363498    2561 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8cb1afc0-048c-4408-bf48-974425ee4843-installation-pull-secrets\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:50:10.363604 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.363510    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgqqd\" (UniqueName: \"kubernetes.io/projected/8cb1afc0-048c-4408-bf48-974425ee4843-kube-api-access-fgqqd\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:50:10.613244 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.613214    2561 generic.go:356] "Generic (PLEG): container finished" podID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerID="f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61" exitCode=0
Jul 31 14:50:10.613424 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.613303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerDied","Data":"f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61"}
Jul 31 14:50:10.616588 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.616516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49" event={"ID":"0db1694b-5738-4896-baa6-7385e719200f","Type":"ContainerStarted","Data":"131c759c6dc43231db185b3a5ccec0eac819db85270e0953bfb6cc88abffed46"}
Jul 31 14:50:10.616768 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.616592    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49" event={"ID":"0db1694b-5738-4896-baa6-7385e719200f","Type":"ContainerStarted","Data":"b25990f75f1bf2b6c095027871473f1cf4fadfdd3924c1293c5b931a61226f5f"}
Jul 31 14:50:10.616768 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.616636    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49" event={"ID":"0db1694b-5738-4896-baa6-7385e719200f","Type":"ContainerStarted","Data":"9a0fa7720f18f2444a9e7a82c62457d5aca448c1b3c7c5329c45d3d71a2837e2"}
Jul 31 14:50:10.616768 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.616691    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:10.617867 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.617843    2561 generic.go:356] "Generic (PLEG): container finished" podID="8cb1afc0-048c-4408-bf48-974425ee4843" containerID="deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d" exitCode=0
Jul 31 14:50:10.617977 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.617905    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"
Jul 31 14:50:10.618040 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.617908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv" event={"ID":"8cb1afc0-048c-4408-bf48-974425ee4843","Type":"ContainerDied","Data":"deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d"}
Jul 31 14:50:10.618087 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.618031    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv" event={"ID":"8cb1afc0-048c-4408-bf48-974425ee4843","Type":"ContainerDied","Data":"b935fa36526c607c060be11cf66ca9604281c115ca97bf4adf54942cd930116b"}
Jul 31 14:50:10.618087 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.618056    2561 scope.go:117] "RemoveContainer" containerID="deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d"
Jul 31 14:50:10.619571 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.619545    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8" event={"ID":"51073248-1761-40d8-82b9-d54cbac2e7e7","Type":"ContainerStarted","Data":"99e7c43767709935145adc7306ce70a413d3035c3b95b6d7e28736e4d62f46a8"}
Jul 31 14:50:10.619793 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.619768    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:10.625828 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.625674    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8"
Jul 31 14:50:10.626939 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.626922    2561 scope.go:117] "RemoveContainer" containerID="deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d"
Jul 31 14:50:10.627210 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:50:10.627187    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d\": container with ID starting with deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d not found: ID does not exist" containerID="deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d"
Jul 31 14:50:10.627301 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.627216    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d"} err="failed to get container status \"deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d\": rpc error: code = NotFound desc = could not find container \"deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d\": container with ID starting with deecb889743549438488d60299f72abefb515d4d2b2f7ec16c38508f007f280d not found: ID does not exist"
Jul 31 14:50:10.665496 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.665439    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49" podStartSLOduration=2.650629747 podStartE2EDuration="8.665418931s" totalImagesPullingTime="6.014789184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:10.665052816 +0000 UTC m=+107.184476854" watchObservedRunningTime="2026-07-31 14:50:10.665418931 +0000 UTC m=+107.184842976"
Jul 31 14:50:10.685740 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.685664    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-6dffbbd67-t6zk8" podStartSLOduration=2.667587282 podStartE2EDuration="6.685650247s" totalImagesPullingTime="4.018062965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:10.684323685 +0000 UTC m=+107.203747736" watchObservedRunningTime="2026-07-31 14:50:10.685650247 +0000 UTC m=+107.205074373"
Jul 31 14:50:10.700038 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.700013    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"]
Jul 31 14:50:10.705835 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:10.705809    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-5b7f7fbdcd-tpjfv"]
Jul 31 14:50:11.632446 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:11.632411    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-5fc555f85b-lnp49"
Jul 31 14:50:11.962505 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:11.962396    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cb1afc0-048c-4408-bf48-974425ee4843" path="/var/lib/kubelet/pods/8cb1afc0-048c-4408-bf48-974425ee4843/volumes"
Jul 31 14:50:15.644980 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:15.644940    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerStarted","Data":"8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d"}
Jul 31 14:50:15.645485 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:15.645034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerStarted","Data":"9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42"}
Jul 31 14:50:15.645485 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:15.645063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerStarted","Data":"be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431"}
Jul 31 14:50:15.645485 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:15.645093    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerStarted","Data":"0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b"}
Jul 31 14:50:15.645485 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:15.645118    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerStarted","Data":"5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38"}
Jul 31 14:50:15.645485 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:15.645151    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerStarted","Data":"71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a"}
Jul 31 14:50:15.675304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:15.675234    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.325715202 podStartE2EDuration="9.675209589s" totalImagesPullingTime="7.333082031s" totalInitContainerRuntime="16.412356ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:50:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:15.673595006 +0000 UTC m=+112.193019044" watchObservedRunningTime="2026-07-31 14:50:15.675209589 +0000 UTC m=+112.194633628"
Jul 31 14:50:16.578136 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:16.578102    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:50:24.813714 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:24.813684    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:24.814141 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:24.813737    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:33.884622 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:33.884586    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:50:33.886981 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:33.886955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bcd6660e-bbc4-47a2-94aa-d65b15d96247-metrics-certs\") pod \"network-metrics-daemon-qpj79\" (UID: \"bcd6660e-bbc4-47a2-94aa-d65b15d96247\") " pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:50:34.071305 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:34.071277    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-jrzg7\""
Jul 31 14:50:34.079306 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:34.079283    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qpj79"
Jul 31 14:50:34.283389 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:34.283359    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qpj79"]
Jul 31 14:50:34.287441 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:50:34.287417    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbcd6660e_bbc4_47a2_94aa_d65b15d96247.slice/crio-d8f59e3419dad0dcfb2ac2018c8b736908d39ca6f44a0b528a4709dbc0688f84 WatchSource:0}: Error finding container d8f59e3419dad0dcfb2ac2018c8b736908d39ca6f44a0b528a4709dbc0688f84: Status 404 returned error can't find the container with id d8f59e3419dad0dcfb2ac2018c8b736908d39ca6f44a0b528a4709dbc0688f84
Jul 31 14:50:34.720074 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:34.720034    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpj79" event={"ID":"bcd6660e-bbc4-47a2-94aa-d65b15d96247","Type":"ContainerStarted","Data":"d8f59e3419dad0dcfb2ac2018c8b736908d39ca6f44a0b528a4709dbc0688f84"}
Jul 31 14:50:36.730934 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:36.730902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpj79" event={"ID":"bcd6660e-bbc4-47a2-94aa-d65b15d96247","Type":"ContainerStarted","Data":"c61985791d91d9b14dd87884b9bcfb393e99b5f347c5657f8711f3f9d250d7ab"}
Jul 31 14:50:36.731357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:36.730960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qpj79" event={"ID":"bcd6660e-bbc4-47a2-94aa-d65b15d96247","Type":"ContainerStarted","Data":"0fe6b4a66e8943b662668499de282bc6273a284a6cffabccdab299a8e0950ee0"}
Jul 31 14:50:36.753766 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:36.753716    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qpj79" podStartSLOduration=131.229340448 podStartE2EDuration="2m12.753699502s" totalImagesPullingTime="1.524359054s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:48:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:50:36.752652699 +0000 UTC m=+133.272076741" watchObservedRunningTime="2026-07-31 14:50:36.753699502 +0000 UTC m=+133.273123539"
Jul 31 14:50:44.820890 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:44.820858    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:44.824757 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:44.824735    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-57bf66bc-klhxq"
Jul 31 14:50:49.786447 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:49.786411    2561 generic.go:356] "Generic (PLEG): container finished" podID="bb00f3ae-bc95-452c-b977-0a857611efcd" containerID="8aa5f3e425b6a629ebb1d64179fc0807ac4e5c2172f0e8f00999217d1e0cce65" exitCode=0
Jul 31 14:50:49.786447 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:49.786450    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-cx286" event={"ID":"bb00f3ae-bc95-452c-b977-0a857611efcd","Type":"ContainerDied","Data":"8aa5f3e425b6a629ebb1d64179fc0807ac4e5c2172f0e8f00999217d1e0cce65"}
Jul 31 14:50:49.786846 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:49.786775    2561 scope.go:117] "RemoveContainer" containerID="8aa5f3e425b6a629ebb1d64179fc0807ac4e5c2172f0e8f00999217d1e0cce65"
Jul 31 14:50:50.792107 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:50:50.792073    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-77696685b-cx286" event={"ID":"bb00f3ae-bc95-452c-b977-0a857611efcd","Type":"ContainerStarted","Data":"e30618d959dc3c17f5d5400e4505f4481649b5f6269bb45e7c86316d7fd93d75"}
Jul 31 14:51:06.577369 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:06.577327    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:06.597421 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:06.597390    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:06.877617 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:06.877530    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:24.627031 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.626946    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:24.627570 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.627278    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="prometheus" containerID="cri-o://71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a" gracePeriod=600
Jul 31 14:51:24.627570 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.627307    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy" containerID="cri-o://9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42" gracePeriod=600
Jul 31 14:51:24.627570 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.627318    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="thanos-sidecar" containerID="cri-o://0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b" gracePeriod=600
Jul 31 14:51:24.627570 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.627345    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy-web" containerID="cri-o://be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431" gracePeriod=600
Jul 31 14:51:24.627570 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.627373    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="config-reloader" containerID="cri-o://5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38" gracePeriod=600
Jul 31 14:51:24.627570 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.627328    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy-thanos" containerID="cri-o://8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d" gracePeriod=600
Jul 31 14:51:24.937960 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.937868    2561 generic.go:356] "Generic (PLEG): container finished" podID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerID="8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d" exitCode=0
Jul 31 14:51:24.937960 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.937895    2561 generic.go:356] "Generic (PLEG): container finished" podID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerID="9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42" exitCode=0
Jul 31 14:51:24.937960 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.937902    2561 generic.go:356] "Generic (PLEG): container finished" podID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerID="0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b" exitCode=0
Jul 31 14:51:24.937960 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.937908    2561 generic.go:356] "Generic (PLEG): container finished" podID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerID="5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38" exitCode=0
Jul 31 14:51:24.937960 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.937913    2561 generic.go:356] "Generic (PLEG): container finished" podID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerID="71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a" exitCode=0
Jul 31 14:51:24.937960 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.937941    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerDied","Data":"8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d"}
Jul 31 14:51:24.938274 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.938046    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerDied","Data":"9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42"}
Jul 31 14:51:24.938274 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.938076    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerDied","Data":"0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b"}
Jul 31 14:51:24.938274 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.938100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerDied","Data":"5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38"}
Jul 31 14:51:24.938274 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:24.938122    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerDied","Data":"71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a"}
Jul 31 14:51:25.877780 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.877753    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:25.946035 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.946003    2561 generic.go:356] "Generic (PLEG): container finished" podID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerID="be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431" exitCode=0
Jul 31 14:51:25.946218 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.946093    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:25.946218 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.946090    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerDied","Data":"be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431"}
Jul 31 14:51:25.946309 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.946217    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"191f86a3-114a-4ab5-87db-0fa795a047cc","Type":"ContainerDied","Data":"578ff55bbc991aba6db5ef518ad9738d4042799f218c644de9bfe384151f5599"}
Jul 31 14:51:25.946309 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.946275    2561 scope.go:117] "RemoveContainer" containerID="8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d"
Jul 31 14:51:25.954851 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.954826    2561 scope.go:117] "RemoveContainer" containerID="9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42"
Jul 31 14:51:25.957121 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957099    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cj2n6\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-kube-api-access-cj2n6\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.957250 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957150    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-kube-rbac-proxy\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.957250 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957202    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-grpc-tls\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957550    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-tls\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957613    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-tls-assets\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957645    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-config\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957669    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-thanos-prometheus-http-client-file\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957705    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957732    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-db\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957758    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-1\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957788    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-metrics-client-certs\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957830    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-0\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957867    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-2\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957906    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-trusted-ca-bundle\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.957959    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-web-config\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.958007    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-config-out\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.958036    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-kubelet-serving-ca-bundle\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.958067    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-metrics-client-ca\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.958103    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-serving-certs-ca-bundle\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.958138    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"191f86a3-114a-4ab5-87db-0fa795a047cc\" (UID: \"191f86a3-114a-4ab5-87db-0fa795a047cc\") "
Jul 31 14:51:25.958813 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.958348    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:25.959795 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.958473    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:25.960471 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.960186    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:25.961107 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.960932    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:25.961107 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.961062    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:25.961796 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.961734    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.961893 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.961809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-kube-api-access-cj2n6" (OuterVolumeSpecName: "kube-api-access-cj2n6") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "kube-api-access-cj2n6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:51:25.961893 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.961866    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.962075 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.962031    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:25.962396 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.962247    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:25.962585 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.962548    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:51:25.962585 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.962571    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.962820 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.962780    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 14:51:25.963527 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.963418    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.963527 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.963484    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.963738 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.963540    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.963738 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.963641    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:51:25.963969 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.963951    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-config-out" (OuterVolumeSpecName: "config-out") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:51:25.964223 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.964199    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-config" (OuterVolumeSpecName: "config") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.964400 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.964382    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.977279 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.977244    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-web-config" (OuterVolumeSpecName: "web-config") pod "191f86a3-114a-4ab5-87db-0fa795a047cc" (UID: "191f86a3-114a-4ab5-87db-0fa795a047cc"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 14:51:25.982135 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.982115    2561 scope.go:117] "RemoveContainer" containerID="be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431"
Jul 31 14:51:25.989524 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.989504    2561 scope.go:117] "RemoveContainer" containerID="0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b"
Jul 31 14:51:25.996595 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:25.996572    2561 scope.go:117] "RemoveContainer" containerID="5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38"
Jul 31 14:51:26.003883 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.003862    2561 scope.go:117] "RemoveContainer" containerID="71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a"
Jul 31 14:51:26.013308 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.013283    2561 scope.go:117] "RemoveContainer" containerID="f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61"
Jul 31 14:51:26.021470 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.021445    2561 scope.go:117] "RemoveContainer" containerID="8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d"
Jul 31 14:51:26.021836 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:51:26.021815    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d\": container with ID starting with 8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d not found: ID does not exist" containerID="8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d"
Jul 31 14:51:26.021900 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.021848    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d"} err="failed to get container status \"8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d\": rpc error: code = NotFound desc = could not find container \"8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d\": container with ID starting with 8538ae6e6156f1e2b5a6a755bc14b51465bb372690ebf03ef83369a4387a183d not found: ID does not exist"
Jul 31 14:51:26.021900 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.021870    2561 scope.go:117] "RemoveContainer" containerID="9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42"
Jul 31 14:51:26.022329 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:51:26.022298    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42\": container with ID starting with 9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42 not found: ID does not exist" containerID="9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42"
Jul 31 14:51:26.022419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.022334    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42"} err="failed to get container status \"9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42\": rpc error: code = NotFound desc = could not find container \"9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42\": container with ID starting with 9533ca704210284f472b48fda72117ab8611f91d78332ef0243f33a0d4f60b42 not found: ID does not exist"
Jul 31 14:51:26.022419 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.022355    2561 scope.go:117] "RemoveContainer" containerID="be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431"
Jul 31 14:51:26.022661 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:51:26.022645    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431\": container with ID starting with be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431 not found: ID does not exist" containerID="be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431"
Jul 31 14:51:26.022711 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.022667    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431"} err="failed to get container status \"be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431\": rpc error: code = NotFound desc = could not find container \"be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431\": container with ID starting with be80105d6ad8df124dd86af19f65064fd0bf54c03328b76d1b2b30d7a1e58431 not found: ID does not exist"
Jul 31 14:51:26.022711 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.022682    2561 scope.go:117] "RemoveContainer" containerID="0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b"
Jul 31 14:51:26.022957 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:51:26.022938    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b\": container with ID starting with 0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b not found: ID does not exist" containerID="0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b"
Jul 31 14:51:26.023018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.022966    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b"} err="failed to get container status \"0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b\": rpc error: code = NotFound desc = could not find container \"0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b\": container with ID starting with 0a85d58fb8f49e8345cce29530b805b7b6134f079b57c511ab2270f9c154e48b not found: ID does not exist"
Jul 31 14:51:26.023018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.022988    2561 scope.go:117] "RemoveContainer" containerID="5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38"
Jul 31 14:51:26.023289 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:51:26.023264    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38\": container with ID starting with 5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38 not found: ID does not exist" containerID="5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38"
Jul 31 14:51:26.023339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.023299    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38"} err="failed to get container status \"5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38\": rpc error: code = NotFound desc = could not find container \"5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38\": container with ID starting with 5f236056e39f74f5557621d57584ae8352b27da0b0c4338b1d8cde87a2079c38 not found: ID does not exist"
Jul 31 14:51:26.023339 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.023321    2561 scope.go:117] "RemoveContainer" containerID="71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a"
Jul 31 14:51:26.023600 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:51:26.023581    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a\": container with ID starting with 71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a not found: ID does not exist" containerID="71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a"
Jul 31 14:51:26.023649 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.023606    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a"} err="failed to get container status \"71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a\": rpc error: code = NotFound desc = could not find container \"71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a\": container with ID starting with 71b8fda6e20150415158292ecf9a601887bcefd4cd9b7633afe97e7b8c38933a not found: ID does not exist"
Jul 31 14:51:26.023649 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.023627    2561 scope.go:117] "RemoveContainer" containerID="f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61"
Jul 31 14:51:26.023895 ip-10-0-129-192 kubenswrapper[2561]: E0731 14:51:26.023867    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61\": container with ID starting with f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61 not found: ID does not exist" containerID="f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61"
Jul 31 14:51:26.023938 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.023907    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61"} err="failed to get container status \"f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61\": rpc error: code = NotFound desc = could not find container \"f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61\": container with ID starting with f2bc7c378d3047c9066d0dd846d3081627c50f575ff37131940e144188edac61 not found: ID does not exist"
Jul 31 14:51:26.059116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059081    2561 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-config-out\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059113    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059124    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-metrics-client-ca\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059134    2561 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059144    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059178    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj2n6\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-kube-api-access-cj2n6\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059188    2561 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-kube-rbac-proxy\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059196    2561 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-grpc-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059206    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059214    2561 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/191f86a3-114a-4ab5-87db-0fa795a047cc-tls-assets\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059221    2561 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059229    2561 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059239    2561 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059247    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-db\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059256    2561 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-secret-metrics-client-certs\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059264    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059276    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059285    2561 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/191f86a3-114a-4ab5-87db-0fa795a047cc-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.059357 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.059294    2561 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/191f86a3-114a-4ab5-87db-0fa795a047cc-web-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:51:26.272885 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.272852    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:26.277822 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.277793    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:26.308952 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.308916    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:26.309251 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309238    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy"
Jul 31 14:51:26.309304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309253    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy"
Jul 31 14:51:26.309304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309263    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy-thanos"
Jul 31 14:51:26.309304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309270    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy-thanos"
Jul 31 14:51:26.309304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309282    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="config-reloader"
Jul 31 14:51:26.309304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309289    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="config-reloader"
Jul 31 14:51:26.309304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309299    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="init-config-reloader"
Jul 31 14:51:26.309304 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309304    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="init-config-reloader"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309315    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="thanos-sidecar"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309320    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="thanos-sidecar"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309326    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cb1afc0-048c-4408-bf48-974425ee4843" containerName="registry"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cb1afc0-048c-4408-bf48-974425ee4843" containerName="registry"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309339    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy-web"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309344    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy-web"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309351    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="prometheus"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309356    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="prometheus"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309401    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy-thanos"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309409    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="thanos-sidecar"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309415    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy-web"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309422    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="config-reloader"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309428    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="kube-rbac-proxy"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309433    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cb1afc0-048c-4408-bf48-974425ee4843" containerName="registry"
Jul 31 14:51:26.309504 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.309439    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" containerName="prometheus"
Jul 31 14:51:26.312641 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.312623    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.315282 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.315258    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 31 14:51:26.315418 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.315319    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 31 14:51:26.315418 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.315326    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 31 14:51:26.315524 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.315460    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 31 14:51:26.315830 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.315807    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 31 14:51:26.315830 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.315810    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-e47enl3gdkn82\""
Jul 31 14:51:26.316018 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.315999    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 31 14:51:26.316080 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.316014    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 31 14:51:26.316133 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.316075    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-4zh25\""
Jul 31 14:51:26.316371 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.316357    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 31 14:51:26.316425 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.316414    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 31 14:51:26.316589 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.316571    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 31 14:51:26.316735 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.316720    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 31 14:51:26.316979 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.316959    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 31 14:51:26.319510 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.319489    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 31 14:51:26.322401 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.322382    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 31 14:51:26.333428 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.333391    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:26.463296 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463256    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463296 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463292    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463536 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463536 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463374    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463536 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463414    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463536 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463454    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463536 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463471    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463536 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463490    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463536 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463773 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463594    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463773 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463620    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463773 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463655    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463773 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463680    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463773 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463707    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-config\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463773 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463733    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.463773 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463764    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.464012 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463779    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fncsx\" (UniqueName: \"kubernetes.io/projected/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-kube-api-access-fncsx\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.464012 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463808    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.464012 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463834    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.464012 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.463854    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565222 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565118    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565222 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565174    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fncsx\" (UniqueName: \"kubernetes.io/projected/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-kube-api-access-fncsx\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565222 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565206    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565466 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565229    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565466 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565250    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565466 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565284    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565466 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565421    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565654 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565464    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565654 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565505    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565654 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565541    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565654 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565596    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565654 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565638    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565889 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565680    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565889 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565711    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565889 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565744    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565889 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565889 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565825    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.565889 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565849    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.566190 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565890    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-config\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.566190 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565905    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.566190 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.565922    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.566190 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.566080    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.566463 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.566441    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.568371 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.567793    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.568546 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.568514    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.568700 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.568671    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.568761 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.568724    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.568901 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.568875    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.569147 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.569114    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-web-config\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.569252 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.569207    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.569413 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.569353    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.569753 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.569731    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.569901 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.569838    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-config-out\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.570047 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.570025    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.570515 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.570494    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.571298 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.571278    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.571392 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.571304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-config\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.571591 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.571574    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.571687 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.571672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.574475 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.574455    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fncsx\" (UniqueName: \"kubernetes.io/projected/9d30abdf-c7b4-4dbf-a2f4-e632fb16914d-kube-api-access-fncsx\") pod \"prometheus-k8s-0\" (UID: \"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.623632 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.623595    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:51:26.777634 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.777598    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 31 14:51:26.780762 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:51:26.780732    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d30abdf_c7b4_4dbf_a2f4_e632fb16914d.slice/crio-fdc26b64a99471ed086313c21d605db094d9b2215a29754993cb6058ae194aa9 WatchSource:0}: Error finding container fdc26b64a99471ed086313c21d605db094d9b2215a29754993cb6058ae194aa9: Status 404 returned error can't find the container with id fdc26b64a99471ed086313c21d605db094d9b2215a29754993cb6058ae194aa9
Jul 31 14:51:26.953063 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.953018    2561 generic.go:356] "Generic (PLEG): container finished" podID="9d30abdf-c7b4-4dbf-a2f4-e632fb16914d" containerID="22abaff3245b4480a0cf8a6a786ad32ff0bdae823f1a83a67520390bc5091ec1" exitCode=0
Jul 31 14:51:26.953521 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.953084    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d","Type":"ContainerDied","Data":"22abaff3245b4480a0cf8a6a786ad32ff0bdae823f1a83a67520390bc5091ec1"}
Jul 31 14:51:26.953521 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:26.953177    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d","Type":"ContainerStarted","Data":"fdc26b64a99471ed086313c21d605db094d9b2215a29754993cb6058ae194aa9"}
Jul 31 14:51:27.962056 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:27.962018    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="191f86a3-114a-4ab5-87db-0fa795a047cc" path="/var/lib/kubelet/pods/191f86a3-114a-4ab5-87db-0fa795a047cc/volumes"
Jul 31 14:51:27.962760 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:27.962734    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d","Type":"ContainerStarted","Data":"414d576b5c6402f2e23dda59d8cf853fe5a32da05b16fe93f42dcd2df1b86e63"}
Jul 31 14:51:27.962872 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:27.962819    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d","Type":"ContainerStarted","Data":"b001b27ca2df06780df1c3dda13cdd6a09d9575a8101023dedd19f5a836ff41a"}
Jul 31 14:51:27.962872 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:27.962865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d","Type":"ContainerStarted","Data":"7f708bc52d82db5116449589f06834fabe3c1c2a5c97761b37f785beec53adc2"}
Jul 31 14:51:27.962980 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:27.962915    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d","Type":"ContainerStarted","Data":"0d1fc1cf683231012add437b87304ac592aafe632f96af2b17812d357f545126"}
Jul 31 14:51:27.962980 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:27.962960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d","Type":"ContainerStarted","Data":"adaef817fcd32d447dcd722a83a84c21142347c2334b591472419996d41cb380"}
Jul 31 14:51:27.963074 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:27.963010    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"9d30abdf-c7b4-4dbf-a2f4-e632fb16914d","Type":"ContainerStarted","Data":"1ed519e6a71e6e5a6055bf5d7223679e3626b21e690fc0f75952955ec7dee360"}
Jul 31 14:51:27.995727 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:27.995670    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=1.975305584 podStartE2EDuration="1.995649975s" totalImagesPullingTime="0s" totalInitContainerRuntime="20.344391ms" isStatefulPod=false podCreationTimestamp="2026-07-31 14:51:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:51:27.992584382 +0000 UTC m=+184.512008420" watchObservedRunningTime="2026-07-31 14:51:27.995649975 +0000 UTC m=+184.515074013"
Jul 31 14:51:31.624320 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:51:31.624271    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:52:26.624557 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:52:26.624517    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:52:26.640822 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:52:26.640786    2561 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:52:27.220775 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:52:27.220743    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 31 14:53:23.878775 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:53:23.878740    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 14:53:23.882735 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:53:23.882710    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 14:53:23.890245 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:53:23.890220    2561 kubelet.go:1648] "Image garbage collection succeeded"
Jul 31 14:57:03.206370 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.206335    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-init-szhr4"]
Jul 31 14:57:03.209658 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.209641    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-szhr4"
Jul 31 14:57:03.212036 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.212009    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 14:57:03.212245 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.212117    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 14:57:03.212245 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.212221    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z6qcv\""
Jul 31 14:57:03.212353 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.212314    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 31 14:57:03.216411 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.216389    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-szhr4"]
Jul 31 14:57:03.267979 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.267943    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnsvn\" (UniqueName: \"kubernetes.io/projected/6ded908c-471d-4137-b304-ecc4d44d8c69-kube-api-access-bnsvn\") pod \"s3-init-szhr4\" (UID: \"6ded908c-471d-4137-b304-ecc4d44d8c69\") " pod="kserve/s3-init-szhr4"
Jul 31 14:57:03.268183 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.268016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ded908c-471d-4137-b304-ecc4d44d8c69-model-cache\") pod \"s3-init-szhr4\" (UID: \"6ded908c-471d-4137-b304-ecc4d44d8c69\") " pod="kserve/s3-init-szhr4"
Jul 31 14:57:03.369294 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.369249    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnsvn\" (UniqueName: \"kubernetes.io/projected/6ded908c-471d-4137-b304-ecc4d44d8c69-kube-api-access-bnsvn\") pod \"s3-init-szhr4\" (UID: \"6ded908c-471d-4137-b304-ecc4d44d8c69\") " pod="kserve/s3-init-szhr4"
Jul 31 14:57:03.369482 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.369351    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ded908c-471d-4137-b304-ecc4d44d8c69-model-cache\") pod \"s3-init-szhr4\" (UID: \"6ded908c-471d-4137-b304-ecc4d44d8c69\") " pod="kserve/s3-init-szhr4"
Jul 31 14:57:03.369776 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.369753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ded908c-471d-4137-b304-ecc4d44d8c69-model-cache\") pod \"s3-init-szhr4\" (UID: \"6ded908c-471d-4137-b304-ecc4d44d8c69\") " pod="kserve/s3-init-szhr4"
Jul 31 14:57:03.378783 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.378753    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnsvn\" (UniqueName: \"kubernetes.io/projected/6ded908c-471d-4137-b304-ecc4d44d8c69-kube-api-access-bnsvn\") pod \"s3-init-szhr4\" (UID: \"6ded908c-471d-4137-b304-ecc4d44d8c69\") " pod="kserve/s3-init-szhr4"
Jul 31 14:57:03.527960 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.527925    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-szhr4"
Jul 31 14:57:03.663593 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.663563    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-init-szhr4"]
Jul 31 14:57:03.668345 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:57:03.668305    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6ded908c_471d_4137_b304_ecc4d44d8c69.slice/crio-810760508e95179ab2d3a6be8acd356c614ff0ca0e54170bcb5940ce4d7fcff3 WatchSource:0}: Error finding container 810760508e95179ab2d3a6be8acd356c614ff0ca0e54170bcb5940ce4d7fcff3: Status 404 returned error can't find the container with id 810760508e95179ab2d3a6be8acd356c614ff0ca0e54170bcb5940ce4d7fcff3
Jul 31 14:57:03.670428 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:03.670410    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 14:57:04.298056 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:04.298018    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-szhr4" event={"ID":"6ded908c-471d-4137-b304-ecc4d44d8c69","Type":"ContainerStarted","Data":"810760508e95179ab2d3a6be8acd356c614ff0ca0e54170bcb5940ce4d7fcff3"}
Jul 31 14:57:07.315809 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:07.315773    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-szhr4" event={"ID":"6ded908c-471d-4137-b304-ecc4d44d8c69","Type":"ContainerStarted","Data":"1a6215523b06bc5aa7d570f67aeabcfbd145a75b76abcdb8ed91b2ee5a31765d"}
Jul 31 14:57:10.329582 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:10.329550    2561 generic.go:356] "Generic (PLEG): container finished" podID="6ded908c-471d-4137-b304-ecc4d44d8c69" containerID="1a6215523b06bc5aa7d570f67aeabcfbd145a75b76abcdb8ed91b2ee5a31765d" exitCode=0
Jul 31 14:57:10.329962 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:10.329614    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-szhr4" event={"ID":"6ded908c-471d-4137-b304-ecc4d44d8c69","Type":"ContainerDied","Data":"1a6215523b06bc5aa7d570f67aeabcfbd145a75b76abcdb8ed91b2ee5a31765d"}
Jul 31 14:57:15.353238 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:15.353133    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-szhr4" event={"ID":"6ded908c-471d-4137-b304-ecc4d44d8c69","Type":"ContainerStarted","Data":"6e2c0783c7c496d4c46681f91fc94c75a72669a84bf2473ca4a51b3e8dc8f52e"}
Jul 31 14:57:15.374863 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:57:15.374800    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-init-szhr4" podStartSLOduration=2.148411846 podStartE2EDuration="12.374777329s" totalImagesPullingTime="8.086972737s" totalInitContainerRuntime="2.139392746s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:57:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:57:15.371930653 +0000 UTC m=+531.891354690" watchObservedRunningTime="2026-07-31 14:57:15.374777329 +0000 UTC m=+531.894201368"
Jul 31 14:58:20.629786 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:20.629754    2561 generic.go:356] "Generic (PLEG): container finished" podID="6ded908c-471d-4137-b304-ecc4d44d8c69" containerID="6e2c0783c7c496d4c46681f91fc94c75a72669a84bf2473ca4a51b3e8dc8f52e" exitCode=0
Jul 31 14:58:20.630215 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:20.629825    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-szhr4" event={"ID":"6ded908c-471d-4137-b304-ecc4d44d8c69","Type":"ContainerDied","Data":"6e2c0783c7c496d4c46681f91fc94c75a72669a84bf2473ca4a51b3e8dc8f52e"}
Jul 31 14:58:21.760713 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:21.760689    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-szhr4"
Jul 31 14:58:21.849042 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:21.849006    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ded908c-471d-4137-b304-ecc4d44d8c69-model-cache\") pod \"6ded908c-471d-4137-b304-ecc4d44d8c69\" (UID: \"6ded908c-471d-4137-b304-ecc4d44d8c69\") "
Jul 31 14:58:21.849271 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:21.849076    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnsvn\" (UniqueName: \"kubernetes.io/projected/6ded908c-471d-4137-b304-ecc4d44d8c69-kube-api-access-bnsvn\") pod \"6ded908c-471d-4137-b304-ecc4d44d8c69\" (UID: \"6ded908c-471d-4137-b304-ecc4d44d8c69\") "
Jul 31 14:58:21.851392 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:21.851355    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ded908c-471d-4137-b304-ecc4d44d8c69-kube-api-access-bnsvn" (OuterVolumeSpecName: "kube-api-access-bnsvn") pod "6ded908c-471d-4137-b304-ecc4d44d8c69" (UID: "6ded908c-471d-4137-b304-ecc4d44d8c69"). InnerVolumeSpecName "kube-api-access-bnsvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:58:21.868431 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:21.868396    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6ded908c-471d-4137-b304-ecc4d44d8c69-model-cache" (OuterVolumeSpecName: "model-cache") pod "6ded908c-471d-4137-b304-ecc4d44d8c69" (UID: "6ded908c-471d-4137-b304-ecc4d44d8c69"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:58:21.950205 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:21.950076    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnsvn\" (UniqueName: \"kubernetes.io/projected/6ded908c-471d-4137-b304-ecc4d44d8c69-kube-api-access-bnsvn\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:58:21.950205 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:21.950121    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6ded908c-471d-4137-b304-ecc4d44d8c69-model-cache\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:58:22.639559 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:22.639528    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-init-szhr4"
Jul 31 14:58:22.639559 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:22.639543    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-init-szhr4" event={"ID":"6ded908c-471d-4137-b304-ecc4d44d8c69","Type":"ContainerDied","Data":"810760508e95179ab2d3a6be8acd356c614ff0ca0e54170bcb5940ce4d7fcff3"}
Jul 31 14:58:22.639761 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:22.639576    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="810760508e95179ab2d3a6be8acd356c614ff0ca0e54170bcb5940ce4d7fcff3"
Jul 31 14:58:23.906838 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:23.906803    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 14:58:23.909434 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:23.909411    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 14:58:33.730624 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.730587    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-custom-b67rd"]
Jul 31 14:58:33.730994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.730888    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ded908c-471d-4137-b304-ecc4d44d8c69" containerName="download-hf-model"
Jul 31 14:58:33.730994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.730899    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ded908c-471d-4137-b304-ecc4d44d8c69" containerName="download-hf-model"
Jul 31 14:58:33.730994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.730914    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ded908c-471d-4137-b304-ecc4d44d8c69" containerName="s3-init"
Jul 31 14:58:33.730994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.730919    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ded908c-471d-4137-b304-ecc4d44d8c69" containerName="s3-init"
Jul 31 14:58:33.730994 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.730976    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ded908c-471d-4137-b304-ecc4d44d8c69" containerName="s3-init"
Jul 31 14:58:33.733310 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.733295    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:33.735597 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.735576    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-custom-artifact\""
Jul 31 14:58:33.735597 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.735588    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 14:58:33.736398 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.736382    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z6qcv\""
Jul 31 14:58:33.736484 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.736383    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 14:58:33.741824 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.741803    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-b67rd"]
Jul 31 14:58:33.853531 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.853497    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-model-cache\") pod \"s3-tls-init-custom-b67rd\" (UID: \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\") " pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:33.853531 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.853535    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj6w2\" (UniqueName: \"kubernetes.io/projected/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-kube-api-access-hj6w2\") pod \"s3-tls-init-custom-b67rd\" (UID: \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\") " pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:33.954724 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.954685    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-model-cache\") pod \"s3-tls-init-custom-b67rd\" (UID: \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\") " pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:33.954901 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.954730    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj6w2\" (UniqueName: \"kubernetes.io/projected/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-kube-api-access-hj6w2\") pod \"s3-tls-init-custom-b67rd\" (UID: \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\") " pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:33.955070 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.955038    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-model-cache\") pod \"s3-tls-init-custom-b67rd\" (UID: \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\") " pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:33.963070 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:33.963042    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj6w2\" (UniqueName: \"kubernetes.io/projected/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-kube-api-access-hj6w2\") pod \"s3-tls-init-custom-b67rd\" (UID: \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\") " pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:34.051828 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:34.051787    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:34.182476 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:34.182453    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-custom-b67rd"]
Jul 31 14:58:34.184461 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:58:34.184433    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda51d69c4_6bfa_4b5b_9fd6_ebb8c057a0d1.slice/crio-f8728146bda0e7a56a7fef567ce95a4cbf46b807943f6ed4d48996d7391ba2ce WatchSource:0}: Error finding container f8728146bda0e7a56a7fef567ce95a4cbf46b807943f6ed4d48996d7391ba2ce: Status 404 returned error can't find the container with id f8728146bda0e7a56a7fef567ce95a4cbf46b807943f6ed4d48996d7391ba2ce
Jul 31 14:58:34.689611 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:34.689577    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-b67rd" event={"ID":"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1","Type":"ContainerStarted","Data":"c4c8ba70b1a1cdfebfcddcccbc6bfa6a469bc022d160ad32cc5ca75c35bf7a33"}
Jul 31 14:58:34.689611 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:34.689613    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-b67rd" event={"ID":"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1","Type":"ContainerStarted","Data":"f8728146bda0e7a56a7fef567ce95a4cbf46b807943f6ed4d48996d7391ba2ce"}
Jul 31 14:58:36.699475 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:36.699440    2561 generic.go:356] "Generic (PLEG): container finished" podID="a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" containerID="c4c8ba70b1a1cdfebfcddcccbc6bfa6a469bc022d160ad32cc5ca75c35bf7a33" exitCode=0
Jul 31 14:58:36.699861 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:36.699516    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-b67rd" event={"ID":"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1","Type":"ContainerDied","Data":"c4c8ba70b1a1cdfebfcddcccbc6bfa6a469bc022d160ad32cc5ca75c35bf7a33"}
Jul 31 14:58:37.706258 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:37.706221    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-b67rd" event={"ID":"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1","Type":"ContainerStarted","Data":"4596ad0ed5311d5b97dee41a74eab8673c43f5b4314532ca7f4b437fbd5a422f"}
Jul 31 14:58:37.724168 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:37.724095    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-custom-b67rd" podStartSLOduration=2.8895545780000003 podStartE2EDuration="4.72407668s" totalImagesPullingTime="0s" totalInitContainerRuntime="1.834522102s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:58:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:58:37.723865232 +0000 UTC m=+614.243289271" watchObservedRunningTime="2026-07-31 14:58:37.72407668 +0000 UTC m=+614.243500718"
Jul 31 14:58:42.730247 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:42.730213    2561 generic.go:356] "Generic (PLEG): container finished" podID="a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" containerID="4596ad0ed5311d5b97dee41a74eab8673c43f5b4314532ca7f4b437fbd5a422f" exitCode=0
Jul 31 14:58:42.730613 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:42.730290    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-b67rd" event={"ID":"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1","Type":"ContainerDied","Data":"4596ad0ed5311d5b97dee41a74eab8673c43f5b4314532ca7f4b437fbd5a422f"}
Jul 31 14:58:43.869087 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:43.869060    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:43.940791 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:43.940761    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hj6w2\" (UniqueName: \"kubernetes.io/projected/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-kube-api-access-hj6w2\") pod \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\" (UID: \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\") "
Jul 31 14:58:43.940976 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:43.940822    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-model-cache\") pod \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\" (UID: \"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1\") "
Jul 31 14:58:43.942890 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:43.942863    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-kube-api-access-hj6w2" (OuterVolumeSpecName: "kube-api-access-hj6w2") pod "a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" (UID: "a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1"). InnerVolumeSpecName "kube-api-access-hj6w2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:58:43.969224 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:43.969192    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-model-cache" (OuterVolumeSpecName: "model-cache") pod "a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" (UID: "a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:58:44.042428 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:44.042392    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hj6w2\" (UniqueName: \"kubernetes.io/projected/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-kube-api-access-hj6w2\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:58:44.042428 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:44.042427    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1-model-cache\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:58:44.740422 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:44.740391    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-custom-b67rd"
Jul 31 14:58:44.740422 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:44.740402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-custom-b67rd" event={"ID":"a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1","Type":"ContainerDied","Data":"f8728146bda0e7a56a7fef567ce95a4cbf46b807943f6ed4d48996d7391ba2ce"}
Jul 31 14:58:44.740622 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:44.740436    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f8728146bda0e7a56a7fef567ce95a4cbf46b807943f6ed4d48996d7391ba2ce"
Jul 31 14:58:47.431224 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.427806    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/s3-tls-init-serving-qrnlt"]
Jul 31 14:58:47.431224 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.428555    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" containerName="download-hf-model"
Jul 31 14:58:47.431224 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.428583    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" containerName="download-hf-model"
Jul 31 14:58:47.431224 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.428604    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" containerName="s3-tls-init-custom"
Jul 31 14:58:47.431224 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.428619    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" containerName="s3-tls-init-custom"
Jul 31 14:58:47.431224 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.428785    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1" containerName="s3-tls-init-custom"
Jul 31 14:58:47.433779 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.433260    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:47.437181 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.437101    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 31 14:58:47.437181 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.437177    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 31 14:58:47.437362 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.437113    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"seaweedfs-tls-serving-artifact\""
Jul 31 14:58:47.438127 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.438106    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-z6qcv\""
Jul 31 14:58:47.440209 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.440186    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-qrnlt"]
Jul 31 14:58:47.570206 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.570143    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7sv8\" (UniqueName: \"kubernetes.io/projected/4117c74d-5256-458c-bf65-e70901742bed-kube-api-access-n7sv8\") pod \"s3-tls-init-serving-qrnlt\" (UID: \"4117c74d-5256-458c-bf65-e70901742bed\") " pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:47.570401 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.570273    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4117c74d-5256-458c-bf65-e70901742bed-model-cache\") pod \"s3-tls-init-serving-qrnlt\" (UID: \"4117c74d-5256-458c-bf65-e70901742bed\") " pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:47.671450 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.671400    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4117c74d-5256-458c-bf65-e70901742bed-model-cache\") pod \"s3-tls-init-serving-qrnlt\" (UID: \"4117c74d-5256-458c-bf65-e70901742bed\") " pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:47.671612 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.671507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7sv8\" (UniqueName: \"kubernetes.io/projected/4117c74d-5256-458c-bf65-e70901742bed-kube-api-access-n7sv8\") pod \"s3-tls-init-serving-qrnlt\" (UID: \"4117c74d-5256-458c-bf65-e70901742bed\") " pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:47.671806 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.671775    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4117c74d-5256-458c-bf65-e70901742bed-model-cache\") pod \"s3-tls-init-serving-qrnlt\" (UID: \"4117c74d-5256-458c-bf65-e70901742bed\") " pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:47.681681 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.681619    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7sv8\" (UniqueName: \"kubernetes.io/projected/4117c74d-5256-458c-bf65-e70901742bed-kube-api-access-n7sv8\") pod \"s3-tls-init-serving-qrnlt\" (UID: \"4117c74d-5256-458c-bf65-e70901742bed\") " pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:47.754176 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.754123    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:47.885032 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:47.885008    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/s3-tls-init-serving-qrnlt"]
Jul 31 14:58:47.887343 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:58:47.887316    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4117c74d_5256_458c_bf65_e70901742bed.slice/crio-68fc3f698bdb7180108de414ad24e5513e2402adca81ff4b98879e4114209016 WatchSource:0}: Error finding container 68fc3f698bdb7180108de414ad24e5513e2402adca81ff4b98879e4114209016: Status 404 returned error can't find the container with id 68fc3f698bdb7180108de414ad24e5513e2402adca81ff4b98879e4114209016
Jul 31 14:58:48.759233 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:48.759195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qrnlt" event={"ID":"4117c74d-5256-458c-bf65-e70901742bed","Type":"ContainerStarted","Data":"cd3eb359d369185c751bbbf2b089e17d68e5b9537b2c77c6592e16f4409bf35a"}
Jul 31 14:58:48.759233 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:48.759237    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qrnlt" event={"ID":"4117c74d-5256-458c-bf65-e70901742bed","Type":"ContainerStarted","Data":"68fc3f698bdb7180108de414ad24e5513e2402adca81ff4b98879e4114209016"}
Jul 31 14:58:50.767981 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:50.767945    2561 generic.go:356] "Generic (PLEG): container finished" podID="4117c74d-5256-458c-bf65-e70901742bed" containerID="cd3eb359d369185c751bbbf2b089e17d68e5b9537b2c77c6592e16f4409bf35a" exitCode=0
Jul 31 14:58:50.768474 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:50.768016    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qrnlt" event={"ID":"4117c74d-5256-458c-bf65-e70901742bed","Type":"ContainerDied","Data":"cd3eb359d369185c751bbbf2b089e17d68e5b9537b2c77c6592e16f4409bf35a"}
Jul 31 14:58:51.773853 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:51.773818    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qrnlt" event={"ID":"4117c74d-5256-458c-bf65-e70901742bed","Type":"ContainerStarted","Data":"a009f26944bb28208869e0c5ec70e3afdb934fa16defe5e247be873ad09fcd4f"}
Jul 31 14:58:51.793582 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:51.793526    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/s3-tls-init-serving-qrnlt" podStartSLOduration=2.782637386 podStartE2EDuration="4.793507956s" totalImagesPullingTime="0s" totalInitContainerRuntime="2.01087057s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:58:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:58:51.792376298 +0000 UTC m=+628.311800337" watchObservedRunningTime="2026-07-31 14:58:51.793507956 +0000 UTC m=+628.312931993"
Jul 31 14:58:56.797200 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:56.797147    2561 generic.go:356] "Generic (PLEG): container finished" podID="4117c74d-5256-458c-bf65-e70901742bed" containerID="a009f26944bb28208869e0c5ec70e3afdb934fa16defe5e247be873ad09fcd4f" exitCode=0
Jul 31 14:58:56.797553 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:56.797218    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qrnlt" event={"ID":"4117c74d-5256-458c-bf65-e70901742bed","Type":"ContainerDied","Data":"a009f26944bb28208869e0c5ec70e3afdb934fa16defe5e247be873ad09fcd4f"}
Jul 31 14:58:57.958451 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:57.958428    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:58.064254 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.064145    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7sv8\" (UniqueName: \"kubernetes.io/projected/4117c74d-5256-458c-bf65-e70901742bed-kube-api-access-n7sv8\") pod \"4117c74d-5256-458c-bf65-e70901742bed\" (UID: \"4117c74d-5256-458c-bf65-e70901742bed\") "
Jul 31 14:58:58.064254 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.064215    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4117c74d-5256-458c-bf65-e70901742bed-model-cache\") pod \"4117c74d-5256-458c-bf65-e70901742bed\" (UID: \"4117c74d-5256-458c-bf65-e70901742bed\") "
Jul 31 14:58:58.066211 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.066181    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4117c74d-5256-458c-bf65-e70901742bed-kube-api-access-n7sv8" (OuterVolumeSpecName: "kube-api-access-n7sv8") pod "4117c74d-5256-458c-bf65-e70901742bed" (UID: "4117c74d-5256-458c-bf65-e70901742bed"). InnerVolumeSpecName "kube-api-access-n7sv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 14:58:58.086969 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.086941    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4117c74d-5256-458c-bf65-e70901742bed-model-cache" (OuterVolumeSpecName: "model-cache") pod "4117c74d-5256-458c-bf65-e70901742bed" (UID: "4117c74d-5256-458c-bf65-e70901742bed"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 14:58:58.165045 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.165016    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7sv8\" (UniqueName: \"kubernetes.io/projected/4117c74d-5256-458c-bf65-e70901742bed-kube-api-access-n7sv8\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:58:58.165045 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.165040    2561 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/4117c74d-5256-458c-bf65-e70901742bed-model-cache\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 14:58:58.806337 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.806303    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/s3-tls-init-serving-qrnlt" event={"ID":"4117c74d-5256-458c-bf65-e70901742bed","Type":"ContainerDied","Data":"68fc3f698bdb7180108de414ad24e5513e2402adca81ff4b98879e4114209016"}
Jul 31 14:58:58.806337 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.806332    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/s3-tls-init-serving-qrnlt"
Jul 31 14:58:58.806337 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:58:58.806340    2561 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68fc3f698bdb7180108de414ad24e5513e2402adca81ff4b98879e4114209016"
Jul 31 14:59:10.792467 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.792433    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"]
Jul 31 14:59:10.794914 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.792838    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4117c74d-5256-458c-bf65-e70901742bed" containerName="s3-tls-init-serving"
Jul 31 14:59:10.794914 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.792854    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4117c74d-5256-458c-bf65-e70901742bed" containerName="s3-tls-init-serving"
Jul 31 14:59:10.794914 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.792869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4117c74d-5256-458c-bf65-e70901742bed" containerName="download-hf-model"
Jul 31 14:59:10.794914 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.792875    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4117c74d-5256-458c-bf65-e70901742bed" containerName="download-hf-model"
Jul 31 14:59:10.794914 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.792939    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4117c74d-5256-458c-bf65-e70901742bed" containerName="s3-tls-init-serving"
Jul 31 14:59:10.795968 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.795954    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.798726 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.798707    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 14:59:10.799119 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.799070    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 14:59:10.799236 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.799127    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-predictor-serving-cert\""
Jul 31 14:59:10.799433 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.799410    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 14:59:10.799433 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.799418    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 14:59:10.799613 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.799426    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-pmml-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 31 14:59:10.799613 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.799468    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 14:59:10.810116 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.810095    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"]
Jul 31 14:59:10.881007 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.880967    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.881230 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.881044    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbpkl\" (UniqueName: \"kubernetes.io/projected/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-api-access-rbpkl\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.881230 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.881086    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.881354 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.881228    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.881354 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.881312    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.981993 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.981950    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.982197 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.982032    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.982197 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.982077    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.982197 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.982119    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbpkl\" (UniqueName: \"kubernetes.io/projected/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-api-access-rbpkl\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.982197 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.982180    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.982650 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.982623    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kserve-provision-location\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.982892 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.982873    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-cabundle-cert\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.982960 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.982897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-rbac-proxy-sar-config\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.984613 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.984586    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-proxy-tls\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:10.994584 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:10.994558    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbpkl\" (UniqueName: \"kubernetes.io/projected/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-api-access-rbpkl\") pod \"isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") " pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:11.106617 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:11.106539    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:11.239835 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:11.239802    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"]
Jul 31 14:59:11.243468 ip-10-0-129-192 kubenswrapper[2561]: W0731 14:59:11.243433    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaac661a4_25ba_4ea7_bdbb_5fd680ab6092.slice/crio-c12539a09c98a6fdd62f55aad92e35363f6e88bb0d3fa1e14417d7e04778511b WatchSource:0}: Error finding container c12539a09c98a6fdd62f55aad92e35363f6e88bb0d3fa1e14417d7e04778511b: Status 404 returned error can't find the container with id c12539a09c98a6fdd62f55aad92e35363f6e88bb0d3fa1e14417d7e04778511b
Jul 31 14:59:11.859141 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:11.859100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" event={"ID":"aac661a4-25ba-4ea7-bdbb-5fd680ab6092","Type":"ContainerStarted","Data":"810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864"}
Jul 31 14:59:11.859568 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:11.859195    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" event={"ID":"aac661a4-25ba-4ea7-bdbb-5fd680ab6092","Type":"ContainerStarted","Data":"c12539a09c98a6fdd62f55aad92e35363f6e88bb0d3fa1e14417d7e04778511b"}
Jul 31 14:59:15.877301 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:15.877265    2561 generic.go:356] "Generic (PLEG): container finished" podID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerID="810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864" exitCode=0
Jul 31 14:59:15.877670 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:15.877339    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" event={"ID":"aac661a4-25ba-4ea7-bdbb-5fd680ab6092","Type":"ContainerDied","Data":"810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864"}
Jul 31 14:59:23.922078 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:23.922032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" event={"ID":"aac661a4-25ba-4ea7-bdbb-5fd680ab6092","Type":"ContainerStarted","Data":"017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692"}
Jul 31 14:59:25.934099 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:25.934063    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" event={"ID":"aac661a4-25ba-4ea7-bdbb-5fd680ab6092","Type":"ContainerStarted","Data":"9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9"}
Jul 31 14:59:25.934527 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:25.934319    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:25.956999 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:25.956935    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podStartSLOduration=3.078882454 podStartE2EDuration="15.956913352s" totalImagesPullingTime="9.199152181s" totalInitContainerRuntime="3.678878717s" isStatefulPod=false podCreationTimestamp="2026-07-31 14:59:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 14:59:25.954886263 +0000 UTC m=+662.474310300" watchObservedRunningTime="2026-07-31 14:59:25.956913352 +0000 UTC m=+662.476337393"
Jul 31 14:59:26.938683 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:26.938651    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:26.939995 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:26.939961    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 14:59:27.943298 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:27.943259    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 14:59:32.948182 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:32.948136    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 14:59:32.948678 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:32.948646    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 14:59:42.948891 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:42.948840    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 14:59:52.949422 ip-10-0-129-192 kubenswrapper[2561]: I0731 14:59:52.949377    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 15:00:02.949211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:02.949150    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 15:00:12.948612 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:12.948568    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 15:00:22.948972 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:22.948879    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 15:00:32.948646 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:32.948604    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 15:00:37.956634 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:37.956596    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 15:00:47.960935 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:47.960905    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 15:00:56.039533 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:56.039496    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"]
Jul 31 15:00:56.040135 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:56.039807    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" containerID="cri-o://017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692" gracePeriod=30
Jul 31 15:00:56.040135 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:56.039844    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kube-rbac-proxy" containerID="cri-o://9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9" gracePeriod=30
Jul 31 15:00:56.318123 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:56.318027    2561 generic.go:356] "Generic (PLEG): container finished" podID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerID="9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9" exitCode=2
Jul 31 15:00:56.318123 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:56.318066    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" event={"ID":"aac661a4-25ba-4ea7-bdbb-5fd680ab6092","Type":"ContainerDied","Data":"9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9"}
Jul 31 15:00:57.944263 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:57.944217    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.27:8643/healthz\": dial tcp 10.133.0.27:8643: connect: connection refused"
Jul 31 15:00:57.957329 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:57.957281    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.27:8080: connect: connection refused"
Jul 31 15:00:59.786182 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.786141    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 15:00:59.830819 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.830789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-proxy-tls\") pod \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") "
Jul 31 15:00:59.830988 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.830834    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-rbac-proxy-sar-config\") pod \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") "
Jul 31 15:00:59.830988 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.830922    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rbpkl\" (UniqueName: \"kubernetes.io/projected/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-api-access-rbpkl\") pod \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") "
Jul 31 15:00:59.830988 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.830968    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kserve-provision-location\") pod \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") "
Jul 31 15:00:59.831164 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.831017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-cabundle-cert\") pod \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\" (UID: \"aac661a4-25ba-4ea7-bdbb-5fd680ab6092\") "
Jul 31 15:00:59.831330 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.831300    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aac661a4-25ba-4ea7-bdbb-5fd680ab6092" (UID: "aac661a4-25ba-4ea7-bdbb-5fd680ab6092"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:00:59.831463 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.831262    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aac661a4-25ba-4ea7-bdbb-5fd680ab6092" (UID: "aac661a4-25ba-4ea7-bdbb-5fd680ab6092"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:00:59.831463 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.831389    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aac661a4-25ba-4ea7-bdbb-5fd680ab6092" (UID: "aac661a4-25ba-4ea7-bdbb-5fd680ab6092"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:00:59.833126 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.833099    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-api-access-rbpkl" (OuterVolumeSpecName: "kube-api-access-rbpkl") pod "aac661a4-25ba-4ea7-bdbb-5fd680ab6092" (UID: "aac661a4-25ba-4ea7-bdbb-5fd680ab6092"). InnerVolumeSpecName "kube-api-access-rbpkl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:00:59.833245 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.833138    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aac661a4-25ba-4ea7-bdbb-5fd680ab6092" (UID: "aac661a4-25ba-4ea7-bdbb-5fd680ab6092"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:00:59.932195 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.932070    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:00:59.932195 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.932104    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:00:59.932195 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.932115    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rbpkl\" (UniqueName: \"kubernetes.io/projected/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kube-api-access-rbpkl\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:00:59.932195 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.932124    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:00:59.932195 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:00:59.932133    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aac661a4-25ba-4ea7-bdbb-5fd680ab6092-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:01:00.335767 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.335725    2561 generic.go:356] "Generic (PLEG): container finished" podID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerID="017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692" exitCode=0
Jul 31 15:01:00.335946 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.335808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" event={"ID":"aac661a4-25ba-4ea7-bdbb-5fd680ab6092","Type":"ContainerDied","Data":"017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692"}
Jul 31 15:01:00.335946 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.335819    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"
Jul 31 15:01:00.335946 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.335917    2561 scope.go:117] "RemoveContainer" containerID="9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9"
Jul 31 15:01:00.336070 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.335902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m" event={"ID":"aac661a4-25ba-4ea7-bdbb-5fd680ab6092","Type":"ContainerDied","Data":"c12539a09c98a6fdd62f55aad92e35363f6e88bb0d3fa1e14417d7e04778511b"}
Jul 31 15:01:00.346361 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.346335    2561 scope.go:117] "RemoveContainer" containerID="017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692"
Jul 31 15:01:00.355398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.355376    2561 scope.go:117] "RemoveContainer" containerID="810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864"
Jul 31 15:01:00.359366 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.359335    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"]
Jul 31 15:01:00.364217 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.364185    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-pmml-v2-kserve-predictor-84cb55486c-g2l7m"]
Jul 31 15:01:00.364735 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.364718    2561 scope.go:117] "RemoveContainer" containerID="9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9"
Jul 31 15:01:00.365319 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:01:00.365299    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9\": container with ID starting with 9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9 not found: ID does not exist" containerID="9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9"
Jul 31 15:01:00.365373 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.365329    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9"} err="failed to get container status \"9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9\": rpc error: code = NotFound desc = could not find container \"9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9\": container with ID starting with 9a2bc2256754010b4541d5841a1c006ea31d882291413a42d01490514a4c64f9 not found: ID does not exist"
Jul 31 15:01:00.365373 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.365349    2561 scope.go:117] "RemoveContainer" containerID="017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692"
Jul 31 15:01:00.365689 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:01:00.365671    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692\": container with ID starting with 017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692 not found: ID does not exist" containerID="017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692"
Jul 31 15:01:00.365731 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.365694    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692"} err="failed to get container status \"017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692\": rpc error: code = NotFound desc = could not find container \"017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692\": container with ID starting with 017032f9e45ea04a430e6f13f9fc3d0067a646adabe402c348409a770a9fd692 not found: ID does not exist"
Jul 31 15:01:00.365731 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.365710    2561 scope.go:117] "RemoveContainer" containerID="810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864"
Jul 31 15:01:00.365952 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:01:00.365936    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864\": container with ID starting with 810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864 not found: ID does not exist" containerID="810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864"
Jul 31 15:01:00.365987 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:00.365957    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864"} err="failed to get container status \"810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864\": rpc error: code = NotFound desc = could not find container \"810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864\": container with ID starting with 810e22ae45faf0dd99fa7d3326576f77721c73d40bb2867109aa1ca174185864 not found: ID does not exist"
Jul 31 15:01:01.961748 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:01:01.961713    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" path="/var/lib/kubelet/pods/aac661a4-25ba-4ea7-bdbb-5fd680ab6092/volumes"
Jul 31 15:03:23.942357 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:23.942277    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:03:23.942946 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:23.942585    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:03:24.813709 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.813669    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"]
Jul 31 15:03:24.814052 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.814038    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="storage-initializer"
Jul 31 15:03:24.814052 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.814053    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="storage-initializer"
Jul 31 15:03:24.814211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.814073    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kube-rbac-proxy"
Jul 31 15:03:24.814211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.814079    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kube-rbac-proxy"
Jul 31 15:03:24.814211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.814087    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container"
Jul 31 15:03:24.814211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.814095    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container"
Jul 31 15:03:24.814211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.814172    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kserve-container"
Jul 31 15:03:24.814211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.814183    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aac661a4-25ba-4ea7-bdbb-5fd680ab6092" containerName="kube-rbac-proxy"
Jul 31 15:03:24.817606 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.817582    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.820052 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.820026    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:03:24.820052 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.820045    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-kube-rbac-proxy-sar-config\""
Jul 31 15:03:24.820271 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.820123    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-predictor-serving-cert\""
Jul 31 15:03:24.820981 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.820937    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:03:24.821129 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.820983    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:03:24.821129 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.821061    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:03:24.821129 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.821089    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:03:24.828770 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.828746    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"]
Jul 31 15:03:24.890094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.890055    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.890313 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.890103    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.890313 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.890213    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.890313 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.890255    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.890440 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.890329    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88qbw\" (UniqueName: \"kubernetes.io/projected/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-api-access-88qbw\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.991099 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.991062    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.991099 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.991104    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.991585 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.991133    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.991585 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.991197    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.991585 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.991262    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88qbw\" (UniqueName: \"kubernetes.io/projected/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-api-access-88qbw\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.991771 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.991745    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kserve-provision-location\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.991932 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.991912    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-cabundle-cert\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.991997 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.991927    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:24.993701 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:24.993672    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-proxy-tls\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:25.001180 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:25.001139    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88qbw\" (UniqueName: \"kubernetes.io/projected/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-api-access-88qbw\") pod \"isvc-predictive-sklearn-predictor-5f849d69df-27c6f\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:25.129441 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:25.129349    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:25.264303 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:25.264268    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"]
Jul 31 15:03:25.267813 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:03:25.267778    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb1df7100_f40e_4a77_89e9_2a8ddfe1c6bc.slice/crio-cabfef2c6c7f782b4666c5a7956353a0ac9e8437b78ebc66bc0488a4b08974fa WatchSource:0}: Error finding container cabfef2c6c7f782b4666c5a7956353a0ac9e8437b78ebc66bc0488a4b08974fa: Status 404 returned error can't find the container with id cabfef2c6c7f782b4666c5a7956353a0ac9e8437b78ebc66bc0488a4b08974fa
Jul 31 15:03:25.935136 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:25.935100    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" event={"ID":"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc","Type":"ContainerStarted","Data":"8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197"}
Jul 31 15:03:25.935433 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:25.935191    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" event={"ID":"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc","Type":"ContainerStarted","Data":"cabfef2c6c7f782b4666c5a7956353a0ac9e8437b78ebc66bc0488a4b08974fa"}
Jul 31 15:03:29.953274 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:29.953236    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerID="8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197" exitCode=0
Jul 31 15:03:29.953648 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:29.953293    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" event={"ID":"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc","Type":"ContainerDied","Data":"8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197"}
Jul 31 15:03:29.954557 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:29.954541    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:03:51.071355 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:51.071305    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" event={"ID":"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc","Type":"ContainerStarted","Data":"5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf"}
Jul 31 15:03:51.071755 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:51.071392    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" event={"ID":"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc","Type":"ContainerStarted","Data":"47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263"}
Jul 31 15:03:51.071755 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:51.071622    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:51.093083 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:51.093028    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podStartSLOduration=2.699337953 podStartE2EDuration="27.092986449s" totalImagesPullingTime="20.657251371s" totalInitContainerRuntime="3.736397125s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:03:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:03:51.092950726 +0000 UTC m=+927.612374766" watchObservedRunningTime="2026-07-31 15:03:51.092986449 +0000 UTC m=+927.612410484"
Jul 31 15:03:52.075311 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:52.075281    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:52.076766 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:52.076738    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:03:53.079618 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:53.079584    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:03:58.084659 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:58.084630    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:03:58.085249 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:03:58.085137    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:04:08.085502 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:04:08.085454    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:04:18.085630 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:04:18.085590    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:04:28.086007 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:04:28.085962    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:04:38.085364 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:04:38.085319    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:04:48.085686 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:04:48.085593    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:04:58.085405 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:04:58.085363    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:05:08.086235 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:08.086206    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:05:14.922831 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:14.922799    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"]
Jul 31 15:05:14.923256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:14.923045    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" containerID="cri-o://47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263" gracePeriod=30
Jul 31 15:05:14.923256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:14.923084    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kube-rbac-proxy" containerID="cri-o://5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf" gracePeriod=30
Jul 31 15:05:15.431245 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:15.431206    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerID="5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf" exitCode=2
Jul 31 15:05:15.431414 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:15.431276    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" event={"ID":"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc","Type":"ContainerDied","Data":"5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf"}
Jul 31 15:05:18.080264 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:18.080216    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.28:8643/healthz\": dial tcp 10.133.0.28:8643: connect: connection refused"
Jul 31 15:05:18.085625 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:18.085600    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.28:8080: connect: connection refused"
Jul 31 15:05:19.863959 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:19.863934    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:05:20.014120 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.014085    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-cabundle-cert\") pod \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") "
Jul 31 15:05:20.014327 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.014125    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-88qbw\" (UniqueName: \"kubernetes.io/projected/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-api-access-88qbw\") pod \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") "
Jul 31 15:05:20.014327 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.014215    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-proxy-tls\") pod \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") "
Jul 31 15:05:20.014327 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.014263    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kserve-provision-location\") pod \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") "
Jul 31 15:05:20.014327 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.014299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-rbac-proxy-sar-config\") pod \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\" (UID: \"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc\") "
Jul 31 15:05:20.014738 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.014676    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" (UID: "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:05:20.014869 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.014809    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" (UID: "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:05:20.015063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.015040    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" (UID: "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:05:20.016642 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.016614    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" (UID: "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:05:20.016852 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.016819    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-api-access-88qbw" (OuterVolumeSpecName: "kube-api-access-88qbw") pod "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" (UID: "b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc"). InnerVolumeSpecName "kube-api-access-88qbw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:05:20.079463 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.079428    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"]
Jul 31 15:05:20.079909 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.079894    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="storage-initializer"
Jul 31 15:05:20.079969 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.079912    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="storage-initializer"
Jul 31 15:05:20.079969 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.079923    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kube-rbac-proxy"
Jul 31 15:05:20.079969 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.079932    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kube-rbac-proxy"
Jul 31 15:05:20.079969 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.079957    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container"
Jul 31 15:05:20.079969 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.079965    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container"
Jul 31 15:05:20.080123 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.080037    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kserve-container"
Jul 31 15:05:20.080123 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.080052    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerName="kube-rbac-proxy"
Jul 31 15:05:20.082720 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.082701    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.084829 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.084808    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-predictor-serving-cert\""
Jul 31 15:05:20.084985 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.084965    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-kube-rbac-proxy-sar-config\""
Jul 31 15:05:20.094546 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.094521    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"]
Jul 31 15:05:20.115607 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.115573    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:05:20.115607 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.115599    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:05:20.115607 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.115613    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:05:20.115817 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.115624    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:05:20.115817 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.115637    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-88qbw\" (UniqueName: \"kubernetes.io/projected/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc-kube-api-access-88qbw\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:05:20.216659 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.216621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-54bzk\" (UniqueName: \"kubernetes.io/projected/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-api-access-54bzk\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.216822 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.216669    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.216822 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.216770    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9a6984-f110-43e5-8471-eeed5e3e049d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.216896 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.216835    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.216930 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.216899    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.317647 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.317543    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9a6984-f110-43e5-8471-eeed5e3e049d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.317647 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.317618    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.317647 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.317657    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.317908 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.317681    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-54bzk\" (UniqueName: \"kubernetes.io/projected/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-api-access-54bzk\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.317908 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:05:20.317798    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-serving-cert: secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 31 15:05:20.317908 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.317813    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.317908 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:05:20.317873    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls podName:dd9a6984-f110-43e5-8471-eeed5e3e049d nodeName:}" failed. No retries permitted until 2026-07-31 15:05:20.817851338 +0000 UTC m=+1017.337275369 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls") pod "isvc-predictive-xgboost-predictor-695f978978-wdcwb" (UID: "dd9a6984-f110-43e5-8471-eeed5e3e049d") : secret "isvc-predictive-xgboost-predictor-serving-cert" not found
Jul 31 15:05:20.318111 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.318089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9a6984-f110-43e5-8471-eeed5e3e049d-kserve-provision-location\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.318405 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.318389    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.318445 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.318429    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-cabundle-cert\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.329363 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.329340    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-54bzk\" (UniqueName: \"kubernetes.io/projected/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-api-access-54bzk\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.454450 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.454420    2561 generic.go:356] "Generic (PLEG): container finished" podID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" containerID="47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263" exitCode=0
Jul 31 15:05:20.454625 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.454461    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" event={"ID":"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc","Type":"ContainerDied","Data":"47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263"}
Jul 31 15:05:20.454625 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.454483    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"
Jul 31 15:05:20.454625 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.454512    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f" event={"ID":"b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc","Type":"ContainerDied","Data":"cabfef2c6c7f782b4666c5a7956353a0ac9e8437b78ebc66bc0488a4b08974fa"}
Jul 31 15:05:20.454625 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.454539    2561 scope.go:117] "RemoveContainer" containerID="5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf"
Jul 31 15:05:20.463793 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.463770    2561 scope.go:117] "RemoveContainer" containerID="47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263"
Jul 31 15:05:20.471116 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.471099    2561 scope.go:117] "RemoveContainer" containerID="8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197"
Jul 31 15:05:20.479083 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.479052    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"]
Jul 31 15:05:20.479665 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.479643    2561 scope.go:117] "RemoveContainer" containerID="5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf"
Jul 31 15:05:20.479954 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:05:20.479936    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf\": container with ID starting with 5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf not found: ID does not exist" containerID="5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf"
Jul 31 15:05:20.480030 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.479967    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf"} err="failed to get container status \"5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf\": rpc error: code = NotFound desc = could not find container \"5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf\": container with ID starting with 5cc1d8c660364243363f23c62be4a94e0721db737a14d64a5ebce2b8d9e5a6bf not found: ID does not exist"
Jul 31 15:05:20.480030 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.479991    2561 scope.go:117] "RemoveContainer" containerID="47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263"
Jul 31 15:05:20.480290 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:05:20.480271    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263\": container with ID starting with 47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263 not found: ID does not exist" containerID="47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263"
Jul 31 15:05:20.480343 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.480297    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263"} err="failed to get container status \"47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263\": rpc error: code = NotFound desc = could not find container \"47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263\": container with ID starting with 47df19f6583fefe56ed2fd65b07db406cb20106183fae28ed367953371c64263 not found: ID does not exist"
Jul 31 15:05:20.480343 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.480314    2561 scope.go:117] "RemoveContainer" containerID="8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197"
Jul 31 15:05:20.480547 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:05:20.480531    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197\": container with ID starting with 8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197 not found: ID does not exist" containerID="8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197"
Jul 31 15:05:20.480591 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.480554    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197"} err="failed to get container status \"8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197\": rpc error: code = NotFound desc = could not find container \"8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197\": container with ID starting with 8b596fce4aaa0cc67c3f481bb9bf2f9b2d589a21ae4006210a0b0ce0eaf26197 not found: ID does not exist"
Jul 31 15:05:20.483473 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.483450    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-predictor-5f849d69df-27c6f"]
Jul 31 15:05:20.821820 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.821786    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.824274 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.824242    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls\") pod \"isvc-predictive-xgboost-predictor-695f978978-wdcwb\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:20.993456 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:20.993418    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:21.126365 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:21.126341    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"]
Jul 31 15:05:21.128526 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:05:21.128501    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd9a6984_f110_43e5_8471_eeed5e3e049d.slice/crio-92b267ed857e72c1730f73411fbbdcf963588e2dc961bfe201779d9ca9e5b066 WatchSource:0}: Error finding container 92b267ed857e72c1730f73411fbbdcf963588e2dc961bfe201779d9ca9e5b066: Status 404 returned error can't find the container with id 92b267ed857e72c1730f73411fbbdcf963588e2dc961bfe201779d9ca9e5b066
Jul 31 15:05:21.466197 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:21.466080    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" event={"ID":"dd9a6984-f110-43e5-8471-eeed5e3e049d","Type":"ContainerStarted","Data":"edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc"}
Jul 31 15:05:21.466197 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:21.466187    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" event={"ID":"dd9a6984-f110-43e5-8471-eeed5e3e049d","Type":"ContainerStarted","Data":"92b267ed857e72c1730f73411fbbdcf963588e2dc961bfe201779d9ca9e5b066"}
Jul 31 15:05:21.961021 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:21.960985    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc" path="/var/lib/kubelet/pods/b1df7100-f40e-4a77-89e9-2a8ddfe1c6bc/volumes"
Jul 31 15:05:25.487578 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:25.487495    2561 generic.go:356] "Generic (PLEG): container finished" podID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerID="edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc" exitCode=0
Jul 31 15:05:25.487927 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:25.487570    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" event={"ID":"dd9a6984-f110-43e5-8471-eeed5e3e049d","Type":"ContainerDied","Data":"edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc"}
Jul 31 15:05:26.496510 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:26.496457    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" event={"ID":"dd9a6984-f110-43e5-8471-eeed5e3e049d","Type":"ContainerStarted","Data":"370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20"}
Jul 31 15:05:26.497030 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:26.496549    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" event={"ID":"dd9a6984-f110-43e5-8471-eeed5e3e049d","Type":"ContainerStarted","Data":"a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f"}
Jul 31 15:05:26.497030 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:26.496774    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:26.517656 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:26.517607    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podStartSLOduration=2.704791246 podStartE2EDuration="6.517590699s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.812799453s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:05:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:05:26.515610247 +0000 UTC m=+1023.035034286" watchObservedRunningTime="2026-07-31 15:05:26.517590699 +0000 UTC m=+1023.037014737"
Jul 31 15:05:27.500886 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:27.500854    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:27.502168 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:27.502127    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:05:28.505016 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:28.504976    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:05:33.509551 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:33.509522    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:05:33.510034 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:33.510002    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:05:43.510205 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:43.510145    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:05:53.510556 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:05:53.510515    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:06:03.510519 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:03.510479    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:06:13.510876 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:13.510832    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:06:23.510867 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:23.510774    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:06:33.510063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:33.510016    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:06:43.510908 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:43.510870    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:06:50.214308 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:50.214274    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"]
Jul 31 15:06:50.214756 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:50.214495    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" containerID="cri-o://a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f" gracePeriod=30
Jul 31 15:06:50.214756 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:50.214556    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kube-rbac-proxy" containerID="cri-o://370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20" gracePeriod=30
Jul 31 15:06:50.865865 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:50.865833    2561 generic.go:356] "Generic (PLEG): container finished" podID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerID="370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20" exitCode=2
Jul 31 15:06:50.866046 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:50.865912    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" event={"ID":"dd9a6984-f110-43e5-8471-eeed5e3e049d","Type":"ContainerDied","Data":"370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20"}
Jul 31 15:06:53.506342 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:53.506290    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.29:8643/healthz\": dial tcp 10.133.0.29:8643: connect: connection refused"
Jul 31 15:06:53.510791 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:53.510762    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.29:8080: connect: connection refused"
Jul 31 15:06:55.161448 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.161424    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:06:55.253303 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.253217    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9a6984-f110-43e5-8471-eeed5e3e049d-kserve-provision-location\") pod \"dd9a6984-f110-43e5-8471-eeed5e3e049d\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") "
Jul 31 15:06:55.253303 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.253284    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-cabundle-cert\") pod \"dd9a6984-f110-43e5-8471-eeed5e3e049d\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") "
Jul 31 15:06:55.253537 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.253306    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-54bzk\" (UniqueName: \"kubernetes.io/projected/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-api-access-54bzk\") pod \"dd9a6984-f110-43e5-8471-eeed5e3e049d\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") "
Jul 31 15:06:55.253537 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.253335    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls\") pod \"dd9a6984-f110-43e5-8471-eeed5e3e049d\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") "
Jul 31 15:06:55.253640 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.253586    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dd9a6984-f110-43e5-8471-eeed5e3e049d-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dd9a6984-f110-43e5-8471-eeed5e3e049d" (UID: "dd9a6984-f110-43e5-8471-eeed5e3e049d"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:06:55.253732 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.253648    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-rbac-proxy-sar-config\") pod \"dd9a6984-f110-43e5-8471-eeed5e3e049d\" (UID: \"dd9a6984-f110-43e5-8471-eeed5e3e049d\") "
Jul 31 15:06:55.253732 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.253685    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dd9a6984-f110-43e5-8471-eeed5e3e049d" (UID: "dd9a6984-f110-43e5-8471-eeed5e3e049d"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:06:55.254007 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.253980    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dd9a6984-f110-43e5-8471-eeed5e3e049d-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:06:55.254127 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.254006    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:06:55.254127 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.254052    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dd9a6984-f110-43e5-8471-eeed5e3e049d" (UID: "dd9a6984-f110-43e5-8471-eeed5e3e049d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:06:55.255855 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.255828    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dd9a6984-f110-43e5-8471-eeed5e3e049d" (UID: "dd9a6984-f110-43e5-8471-eeed5e3e049d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:06:55.255967 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.255912    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-api-access-54bzk" (OuterVolumeSpecName: "kube-api-access-54bzk") pod "dd9a6984-f110-43e5-8471-eeed5e3e049d" (UID: "dd9a6984-f110-43e5-8471-eeed5e3e049d"). InnerVolumeSpecName "kube-api-access-54bzk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:06:55.354782 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.354738    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-54bzk\" (UniqueName: \"kubernetes.io/projected/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-api-access-54bzk\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:06:55.354782 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.354777    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dd9a6984-f110-43e5-8471-eeed5e3e049d-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:06:55.354782 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.354787    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dd9a6984-f110-43e5-8471-eeed5e3e049d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:06:55.371597 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.371565    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"]
Jul 31 15:06:55.371910 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.371898    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kube-rbac-proxy"
Jul 31 15:06:55.371958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.371912    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kube-rbac-proxy"
Jul 31 15:06:55.371958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.371927    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="storage-initializer"
Jul 31 15:06:55.371958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.371933    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="storage-initializer"
Jul 31 15:06:55.371958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.371948    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container"
Jul 31 15:06:55.371958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.371954    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container"
Jul 31 15:06:55.372104 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.372002    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kube-rbac-proxy"
Jul 31 15:06:55.372104 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.372013    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerName="kserve-container"
Jul 31 15:06:55.375356 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.375337    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.378821 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.378565    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-predictor-serving-cert\""
Jul 31 15:06:55.378821 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.378650    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-kube-rbac-proxy-sar-config\""
Jul 31 15:06:55.392380 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.392250    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"]
Jul 31 15:06:55.455707 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.455671    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jz8g\" (UniqueName: \"kubernetes.io/projected/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-api-access-2jz8g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.455912 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.455723    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.455912 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.455795    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.455912 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.455819    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.455912 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.455890    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.557403 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.557310    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.557403 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.557357    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.557403 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.557378    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.557659 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.557413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jz8g\" (UniqueName: \"kubernetes.io/projected/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-api-access-2jz8g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.557659 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.557438    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.557659 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:06:55.557489    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-serving-cert: secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 31 15:06:55.557659 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:06:55.557570    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls podName:8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7 nodeName:}" failed. No retries permitted until 2026-07-31 15:06:56.057549767 +0000 UTC m=+1112.576973791 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls") pod "isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" (UID: "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7") : secret "isvc-predictive-lightgbm-predictor-serving-cert" not found
Jul 31 15:06:55.557890 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.557869    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kserve-provision-location\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.558076 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.558058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-cabundle-cert\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.558118 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.558097    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.566578 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.566555    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jz8g\" (UniqueName: \"kubernetes.io/projected/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-api-access-2jz8g\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:55.889906 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.889815    2561 generic.go:356] "Generic (PLEG): container finished" podID="dd9a6984-f110-43e5-8471-eeed5e3e049d" containerID="a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f" exitCode=0
Jul 31 15:06:55.889906 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.889873    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"
Jul 31 15:06:55.889906 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.889897    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" event={"ID":"dd9a6984-f110-43e5-8471-eeed5e3e049d","Type":"ContainerDied","Data":"a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f"}
Jul 31 15:06:55.890116 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.889971    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb" event={"ID":"dd9a6984-f110-43e5-8471-eeed5e3e049d","Type":"ContainerDied","Data":"92b267ed857e72c1730f73411fbbdcf963588e2dc961bfe201779d9ca9e5b066"}
Jul 31 15:06:55.890116 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.890012    2561 scope.go:117] "RemoveContainer" containerID="370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20"
Jul 31 15:06:55.898752 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.898732    2561 scope.go:117] "RemoveContainer" containerID="a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f"
Jul 31 15:06:55.906084 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.906065    2561 scope.go:117] "RemoveContainer" containerID="edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc"
Jul 31 15:06:55.913887 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.913780    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"]
Jul 31 15:06:55.913950 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.913919    2561 scope.go:117] "RemoveContainer" containerID="370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20"
Jul 31 15:06:55.914243 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:06:55.914226    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20\": container with ID starting with 370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20 not found: ID does not exist" containerID="370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20"
Jul 31 15:06:55.914294 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.914250    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20"} err="failed to get container status \"370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20\": rpc error: code = NotFound desc = could not find container \"370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20\": container with ID starting with 370748fabd451ec4cfb90b692c201282048e211527d7554c97bb29283b64ad20 not found: ID does not exist"
Jul 31 15:06:55.914294 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.914267    2561 scope.go:117] "RemoveContainer" containerID="a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f"
Jul 31 15:06:55.914539 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:06:55.914521    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f\": container with ID starting with a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f not found: ID does not exist" containerID="a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f"
Jul 31 15:06:55.914584 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.914545    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f"} err="failed to get container status \"a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f\": rpc error: code = NotFound desc = could not find container \"a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f\": container with ID starting with a5f8a2884377ba8b1ef74cda3a9eb67dacaa72ab550905e143e97ed104c49f0f not found: ID does not exist"
Jul 31 15:06:55.914584 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.914562    2561 scope.go:117] "RemoveContainer" containerID="edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc"
Jul 31 15:06:55.914824 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:06:55.914807    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc\": container with ID starting with edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc not found: ID does not exist" containerID="edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc"
Jul 31 15:06:55.914880 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.914832    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc"} err="failed to get container status \"edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc\": rpc error: code = NotFound desc = could not find container \"edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc\": container with ID starting with edfc7ce73ad2265dd7b6a698c734b5c20f3f0cd4b15d8b4b50fec465b8c69bcc not found: ID does not exist"
Jul 31 15:06:55.921833 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.921812    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-predictor-695f978978-wdcwb"]
Jul 31 15:06:55.961299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:55.961273    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dd9a6984-f110-43e5-8471-eeed5e3e049d" path="/var/lib/kubelet/pods/dd9a6984-f110-43e5-8471-eeed5e3e049d/volumes"
Jul 31 15:06:56.062598 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:56.062564    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:56.064993 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:56.064968    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls\") pod \"isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:56.290180 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:56.290118    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:06:56.432295 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:56.432257    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"]
Jul 31 15:06:56.436502 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:06:56.436466    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ba1fe48_6ed6_4821_b6a5_a5cef7b598c7.slice/crio-1f19dcd1748a82ef7c52f999f954c5925234e72e3aa9e50837691987ace2ec2b WatchSource:0}: Error finding container 1f19dcd1748a82ef7c52f999f954c5925234e72e3aa9e50837691987ace2ec2b: Status 404 returned error can't find the container with id 1f19dcd1748a82ef7c52f999f954c5925234e72e3aa9e50837691987ace2ec2b
Jul 31 15:06:56.897444 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:56.897409    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" event={"ID":"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7","Type":"ContainerStarted","Data":"64d1b7cd051ccb9ad70e9a1cd07575298460a210729e0f93c60abec568bf2bf3"}
Jul 31 15:06:56.897652 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:06:56.897486    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" event={"ID":"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7","Type":"ContainerStarted","Data":"1f19dcd1748a82ef7c52f999f954c5925234e72e3aa9e50837691987ace2ec2b"}
Jul 31 15:07:00.919183 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:00.919128    2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerID="64d1b7cd051ccb9ad70e9a1cd07575298460a210729e0f93c60abec568bf2bf3" exitCode=0
Jul 31 15:07:00.919623 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:00.919201    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" event={"ID":"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7","Type":"ContainerDied","Data":"64d1b7cd051ccb9ad70e9a1cd07575298460a210729e0f93c60abec568bf2bf3"}
Jul 31 15:07:01.926337 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:01.926306    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" event={"ID":"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7","Type":"ContainerStarted","Data":"0fa33e59d872d80ade872e662766520614bd1f5bb859c7e5bea263f1c7724624"}
Jul 31 15:07:01.926774 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:01.926368    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" event={"ID":"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7","Type":"ContainerStarted","Data":"2195019abac1d0d7adece0d2c74c1bbdd4dfd3e0ce33752a0f12b07c37177171"}
Jul 31 15:07:01.926774 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:01.926564    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:07:01.952049 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:01.951993    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podStartSLOduration=3.030336399 podStartE2EDuration="6.951979328s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.921642929s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:06:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:07:01.949489742 +0000 UTC m=+1118.468913781" watchObservedRunningTime="2026-07-31 15:07:01.951979328 +0000 UTC m=+1118.471403365"
Jul 31 15:07:02.931885 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:02.931844    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:07:02.933327 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:02.933297    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:07:03.942011 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:03.941965    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:07:08.946978 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:08.946947    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:07:08.947634 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:08.947605    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:07:18.948050 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:18.948009    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:07:28.947787 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:28.947747    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:07:38.948407 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:38.948363    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:07:48.947756 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:48.947670    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:07:58.947746 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:07:58.947705    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:08:08.947718 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:08.947673    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:08:18.948328 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:18.948294    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:08:23.972527 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:23.972498    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:08:23.975472 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:23.975446    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:08:25.449040 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:25.449004    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"]
Jul 31 15:08:25.449518 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:25.449275    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" containerID="cri-o://2195019abac1d0d7adece0d2c74c1bbdd4dfd3e0ce33752a0f12b07c37177171" gracePeriod=30
Jul 31 15:08:25.449518 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:25.449376    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kube-rbac-proxy" containerID="cri-o://0fa33e59d872d80ade872e662766520614bd1f5bb859c7e5bea263f1c7724624" gracePeriod=30
Jul 31 15:08:26.291672 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:26.291634    2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerID="0fa33e59d872d80ade872e662766520614bd1f5bb859c7e5bea263f1c7724624" exitCode=2
Jul 31 15:08:26.291887 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:26.291703    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" event={"ID":"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7","Type":"ContainerDied","Data":"0fa33e59d872d80ade872e662766520614bd1f5bb859c7e5bea263f1c7724624"}
Jul 31 15:08:28.942378 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:28.942337    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.30:8643/healthz\": dial tcp 10.133.0.30:8643: connect: connection refused"
Jul 31 15:08:28.947949 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:28.947924    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.30:8080: connect: connection refused"
Jul 31 15:08:30.311690 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.311657    2561 generic.go:356] "Generic (PLEG): container finished" podID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerID="2195019abac1d0d7adece0d2c74c1bbdd4dfd3e0ce33752a0f12b07c37177171" exitCode=0
Jul 31 15:08:30.312051 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.311727    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" event={"ID":"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7","Type":"ContainerDied","Data":"2195019abac1d0d7adece0d2c74c1bbdd4dfd3e0ce33752a0f12b07c37177171"}
Jul 31 15:08:30.630280 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.630246    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"]
Jul 31 15:08:30.633756 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.633733    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.636073 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.636052    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-predictor-serving-cert\""
Jul 31 15:08:30.636073 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.636067    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-sklearn-v2-kube-rbac-proxy-sar-config\""
Jul 31 15:08:30.644049 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.644024    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"]
Jul 31 15:08:30.696864 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.696820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.697077 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.696882    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.697077 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.697017    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.697232 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.697090    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.697232 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.697122    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpk8b\" (UniqueName: \"kubernetes.io/projected/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-api-access-gpk8b\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.794462 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.794435    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:08:30.798580 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.798558    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.798720 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.798587    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.798720 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.798629    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.798720 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.798658    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.798720 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.798677    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpk8b\" (UniqueName: \"kubernetes.io/projected/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-api-access-gpk8b\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.798947 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:08:30.798794    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-serving-cert: secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 31 15:08:30.798947 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:08:30.798851    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls podName:7fbb2d16-3710-4783-b7d0-e8376b9fa26f nodeName:}" failed. No retries permitted until 2026-07-31 15:08:31.298831883 +0000 UTC m=+1207.818255899 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls") pod "isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" (UID: "7fbb2d16-3710-4783-b7d0-e8376b9fa26f") : secret "isvc-predictive-sklearn-v2-predictor-serving-cert" not found
Jul 31 15:08:30.799289 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.799265    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kserve-provision-location\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.799387 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.799362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.799429 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.799362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-cabundle-cert\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.810692 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.810664    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpk8b\" (UniqueName: \"kubernetes.io/projected/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-api-access-gpk8b\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:30.899603 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.899518    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-rbac-proxy-sar-config\") pod \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") "
Jul 31 15:08:30.899603 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.899562    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls\") pod \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") "
Jul 31 15:08:30.899814 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.899615    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kserve-provision-location\") pod \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") "
Jul 31 15:08:30.899814 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.899637    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-cabundle-cert\") pod \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") "
Jul 31 15:08:30.899814 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.899696    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2jz8g\" (UniqueName: \"kubernetes.io/projected/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-api-access-2jz8g\") pod \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\" (UID: \"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7\") "
Jul 31 15:08:30.899975 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.899901    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" (UID: "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:08:30.900039 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.899971    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" (UID: "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:08:30.900095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.900075    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:08:30.900142 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.900094    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:08:30.900142 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.900090    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" (UID: "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:08:30.901762 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.901742    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" (UID: "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:08:30.901847 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:30.901819    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-api-access-2jz8g" (OuterVolumeSpecName: "kube-api-access-2jz8g") pod "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" (UID: "8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7"). InnerVolumeSpecName "kube-api-access-2jz8g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:08:31.001554 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.001519    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:08:31.001554 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.001553    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:08:31.001747 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.001567    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2jz8g\" (UniqueName: \"kubernetes.io/projected/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7-kube-api-access-2jz8g\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:08:31.304735 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.304694    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:31.307147 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.307122    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls\") pod \"isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") " pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:31.323663 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.323638    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"
Jul 31 15:08:31.324063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.323638    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96" event={"ID":"8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7","Type":"ContainerDied","Data":"1f19dcd1748a82ef7c52f999f954c5925234e72e3aa9e50837691987ace2ec2b"}
Jul 31 15:08:31.324063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.323798    2561 scope.go:117] "RemoveContainer" containerID="0fa33e59d872d80ade872e662766520614bd1f5bb859c7e5bea263f1c7724624"
Jul 31 15:08:31.332591 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.332564    2561 scope.go:117] "RemoveContainer" containerID="2195019abac1d0d7adece0d2c74c1bbdd4dfd3e0ce33752a0f12b07c37177171"
Jul 31 15:08:31.340656 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.340635    2561 scope.go:117] "RemoveContainer" containerID="64d1b7cd051ccb9ad70e9a1cd07575298460a210729e0f93c60abec568bf2bf3"
Jul 31 15:08:31.355337 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.355314    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"]
Jul 31 15:08:31.357778 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.357758    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-predictor-ff56bcbb5-h2d96"]
Jul 31 15:08:31.545147 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.545108    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:31.690735 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.690684    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"]
Jul 31 15:08:31.693210 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:08:31.693181    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7fbb2d16_3710_4783_b7d0_e8376b9fa26f.slice/crio-25560d573e6a3b9b56b3d69b2885905dfa64a6502515520a56755cea83016805 WatchSource:0}: Error finding container 25560d573e6a3b9b56b3d69b2885905dfa64a6502515520a56755cea83016805: Status 404 returned error can't find the container with id 25560d573e6a3b9b56b3d69b2885905dfa64a6502515520a56755cea83016805
Jul 31 15:08:31.961809 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:31.961731    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" path="/var/lib/kubelet/pods/8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7/volumes"
Jul 31 15:08:32.331005 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:32.330970    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" event={"ID":"7fbb2d16-3710-4783-b7d0-e8376b9fa26f","Type":"ContainerStarted","Data":"c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2"}
Jul 31 15:08:32.331515 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:32.331033    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" event={"ID":"7fbb2d16-3710-4783-b7d0-e8376b9fa26f","Type":"ContainerStarted","Data":"25560d573e6a3b9b56b3d69b2885905dfa64a6502515520a56755cea83016805"}
Jul 31 15:08:36.350323 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:36.350290    2561 generic.go:356] "Generic (PLEG): container finished" podID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerID="c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2" exitCode=0
Jul 31 15:08:36.350735 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:36.350358    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" event={"ID":"7fbb2d16-3710-4783-b7d0-e8376b9fa26f","Type":"ContainerDied","Data":"c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2"}
Jul 31 15:08:37.357844 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:37.357804    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" event={"ID":"7fbb2d16-3710-4783-b7d0-e8376b9fa26f","Type":"ContainerStarted","Data":"18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211"}
Jul 31 15:08:37.358303 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:37.357892    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" event={"ID":"7fbb2d16-3710-4783-b7d0-e8376b9fa26f","Type":"ContainerStarted","Data":"37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1"}
Jul 31 15:08:37.358303 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:37.358119    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:37.358303 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:37.358271    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:08:37.382436 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:37.382376    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podStartSLOduration=3.49290957 podStartE2EDuration="7.382353309s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.889443739s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:08:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:08:37.381551313 +0000 UTC m=+1213.900975352" watchObservedRunningTime="2026-07-31 15:08:37.382353309 +0000 UTC m=+1213.901777350"
Jul 31 15:08:43.367828 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:08:43.367796    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:09:13.369654 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:09:13.369606    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:09:23.369287 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:09:23.369246    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:09:33.368919 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:09:33.368877    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:09:43.369106 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:09:43.369060    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:09:53.395097 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:09:53.395064    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:10:01.574038 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:01.574003    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"]
Jul 31 15:10:01.574460 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:01.574261    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container" containerID="cri-o://37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1" gracePeriod=30
Jul 31 15:10:01.574460 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:01.574295    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kube-rbac-proxy" containerID="cri-o://18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211" gracePeriod=30
Jul 31 15:10:01.744063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:01.744028    2561 generic.go:356] "Generic (PLEG): container finished" podID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerID="18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211" exitCode=2
Jul 31 15:10:01.744259 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:01.744097    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" event={"ID":"7fbb2d16-3710-4783-b7d0-e8376b9fa26f","Type":"ContainerDied","Data":"18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211"}
Jul 31 15:10:03.363348 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:03.363304    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.31:8643/healthz\": dial tcp 10.133.0.31:8643: connect: connection refused"
Jul 31 15:10:03.368720 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:03.368696    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.31:8080/v2/models/isvc-predictive-sklearn-v2/ready\": dial tcp 10.133.0.31:8080: connect: connection refused"
Jul 31 15:10:06.517209 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.517181    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:10:06.532384 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.532335    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gpk8b\" (UniqueName: \"kubernetes.io/projected/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-api-access-gpk8b\") pod \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") "
Jul 31 15:10:06.532384 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.532394    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls\") pod \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") "
Jul 31 15:10:06.532654 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.532444    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kserve-provision-location\") pod \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") "
Jul 31 15:10:06.532654 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.532482    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-rbac-proxy-sar-config\") pod \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") "
Jul 31 15:10:06.532654 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.532545    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-cabundle-cert\") pod \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\" (UID: \"7fbb2d16-3710-4783-b7d0-e8376b9fa26f\") "
Jul 31 15:10:06.532814 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.532781    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "7fbb2d16-3710-4783-b7d0-e8376b9fa26f" (UID: "7fbb2d16-3710-4783-b7d0-e8376b9fa26f"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:10:06.532899 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.532874    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "7fbb2d16-3710-4783-b7d0-e8376b9fa26f" (UID: "7fbb2d16-3710-4783-b7d0-e8376b9fa26f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:10:06.532989 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.532966    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "7fbb2d16-3710-4783-b7d0-e8376b9fa26f" (UID: "7fbb2d16-3710-4783-b7d0-e8376b9fa26f"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:10:06.534571 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.534540    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-api-access-gpk8b" (OuterVolumeSpecName: "kube-api-access-gpk8b") pod "7fbb2d16-3710-4783-b7d0-e8376b9fa26f" (UID: "7fbb2d16-3710-4783-b7d0-e8376b9fa26f"). InnerVolumeSpecName "kube-api-access-gpk8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:10:06.534571 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.534567    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "7fbb2d16-3710-4783-b7d0-e8376b9fa26f" (UID: "7fbb2d16-3710-4783-b7d0-e8376b9fa26f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:10:06.634029 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.634001    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:10:06.634029 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.634026    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:10:06.634029 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.634037    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:10:06.634269 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.634047    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpk8b\" (UniqueName: \"kubernetes.io/projected/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-kube-api-access-gpk8b\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:10:06.634269 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.634056    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/7fbb2d16-3710-4783-b7d0-e8376b9fa26f-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:10:06.755257 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755223    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"]
Jul 31 15:10:06.755576 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755563    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="storage-initializer"
Jul 31 15:10:06.755620 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755578    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="storage-initializer"
Jul 31 15:10:06.755620 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755589    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container"
Jul 31 15:10:06.755620 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755598    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container"
Jul 31 15:10:06.755620 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755613    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="storage-initializer"
Jul 31 15:10:06.755620 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755620    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="storage-initializer"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755629    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755634    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755640    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kube-rbac-proxy"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755645    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kube-rbac-proxy"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755654    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kube-rbac-proxy"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755660    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kube-rbac-proxy"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755709    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kserve-container"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755718    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerName="kube-rbac-proxy"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755724    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kserve-container"
Jul 31 15:10:06.755773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.755731    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8ba1fe48-6ed6-4821-b6a5-a5cef7b598c7" containerName="kube-rbac-proxy"
Jul 31 15:10:06.758897 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.758876    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.761286 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.761262    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 31 15:10:06.761536 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.761517    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-xgboost-v2-predictor-serving-cert\""
Jul 31 15:10:06.767925 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.767903    2561 generic.go:356] "Generic (PLEG): container finished" podID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" containerID="37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1" exitCode=0
Jul 31 15:10:06.768028 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.767960    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" event={"ID":"7fbb2d16-3710-4783-b7d0-e8376b9fa26f","Type":"ContainerDied","Data":"37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1"}
Jul 31 15:10:06.768028 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.767977    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"
Jul 31 15:10:06.768028 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.768006    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7" event={"ID":"7fbb2d16-3710-4783-b7d0-e8376b9fa26f","Type":"ContainerDied","Data":"25560d573e6a3b9b56b3d69b2885905dfa64a6502515520a56755cea83016805"}
Jul 31 15:10:06.768188 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.768035    2561 scope.go:117] "RemoveContainer" containerID="18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211"
Jul 31 15:10:06.771359 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.771338    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"]
Jul 31 15:10:06.778318 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.778296    2561 scope.go:117] "RemoveContainer" containerID="37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1"
Jul 31 15:10:06.788030 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.788008    2561 scope.go:117] "RemoveContainer" containerID="c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2"
Jul 31 15:10:06.796762 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.796742    2561 scope.go:117] "RemoveContainer" containerID="18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211"
Jul 31 15:10:06.797024 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:10:06.797002    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211\": container with ID starting with 18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211 not found: ID does not exist" containerID="18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211"
Jul 31 15:10:06.797112 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.797036    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211"} err="failed to get container status \"18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211\": rpc error: code = NotFound desc = could not find container \"18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211\": container with ID starting with 18ff2318231b7ed9396ff53239712465532a940af6f7f276cd65ac0cebec9211 not found: ID does not exist"
Jul 31 15:10:06.797112 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.797062    2561 scope.go:117] "RemoveContainer" containerID="37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1"
Jul 31 15:10:06.797366 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:10:06.797349    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1\": container with ID starting with 37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1 not found: ID does not exist" containerID="37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1"
Jul 31 15:10:06.797414 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.797373    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1"} err="failed to get container status \"37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1\": rpc error: code = NotFound desc = could not find container \"37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1\": container with ID starting with 37103944ca3c26e179ca240ee6d59d257ef9b2d0725b3c7a986ef0e4b73698c1 not found: ID does not exist"
Jul 31 15:10:06.797414 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.797392    2561 scope.go:117] "RemoveContainer" containerID="c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2"
Jul 31 15:10:06.797636 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:10:06.797619    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2\": container with ID starting with c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2 not found: ID does not exist" containerID="c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2"
Jul 31 15:10:06.797693 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.797641    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2"} err="failed to get container status \"c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2\": rpc error: code = NotFound desc = could not find container \"c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2\": container with ID starting with c254fe9912d8b1c42961a330279df77ec871dfe742b5f9576c5d00b6f0e934c2 not found: ID does not exist"
Jul 31 15:10:06.802793 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.802772    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"]
Jul 31 15:10:06.808380 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.808360    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-sklearn-v2-predictor-745f9c8c9f-szqr7"]
Jul 31 15:10:06.835968 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.835929    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.835968 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.835972    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.836249 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.836013    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8710c59b-e238-4d8f-a699-2b32c5e56fd0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.836249 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.836045    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.836249 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.836112    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6sk6b\" (UniqueName: \"kubernetes.io/projected/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-api-access-6sk6b\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.937011 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.936966    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6sk6b\" (UniqueName: \"kubernetes.io/projected/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-api-access-6sk6b\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.937235 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.937036    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.937235 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.937075    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.937235 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.937120    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8710c59b-e238-4d8f-a699-2b32c5e56fd0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.937235 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.937196    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.937612 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.937590    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kserve-provision-location\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.937789 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.937770    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.937832 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.937796    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-cabundle-cert\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.939644 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.939628    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8710c59b-e238-4d8f-a699-2b32c5e56fd0-proxy-tls\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:06.946412 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:06.946393    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6sk6b\" (UniqueName: \"kubernetes.io/projected/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-api-access-6sk6b\") pod \"isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") " pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:07.072019 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:07.071866    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:07.210363 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:07.210340    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"]
Jul 31 15:10:07.212256 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:10:07.212229    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8710c59b_e238_4d8f_a699_2b32c5e56fd0.slice/crio-d5d33671c1c27b5ae907e2517919abe95547f7feb07ff300ccffeb60e18f1241 WatchSource:0}: Error finding container d5d33671c1c27b5ae907e2517919abe95547f7feb07ff300ccffeb60e18f1241: Status 404 returned error can't find the container with id d5d33671c1c27b5ae907e2517919abe95547f7feb07ff300ccffeb60e18f1241
Jul 31 15:10:07.781225 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:07.781184    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" event={"ID":"8710c59b-e238-4d8f-a699-2b32c5e56fd0","Type":"ContainerStarted","Data":"5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3"}
Jul 31 15:10:07.781710 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:07.781284    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" event={"ID":"8710c59b-e238-4d8f-a699-2b32c5e56fd0","Type":"ContainerStarted","Data":"d5d33671c1c27b5ae907e2517919abe95547f7feb07ff300ccffeb60e18f1241"}
Jul 31 15:10:07.961512 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:07.961478    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7fbb2d16-3710-4783-b7d0-e8376b9fa26f" path="/var/lib/kubelet/pods/7fbb2d16-3710-4783-b7d0-e8376b9fa26f/volumes"
Jul 31 15:10:11.801468 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:11.801436    2561 generic.go:356] "Generic (PLEG): container finished" podID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerID="5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3" exitCode=0
Jul 31 15:10:11.801846 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:11.801482    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" event={"ID":"8710c59b-e238-4d8f-a699-2b32c5e56fd0","Type":"ContainerDied","Data":"5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3"}
Jul 31 15:10:12.808105 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:12.808072    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" event={"ID":"8710c59b-e238-4d8f-a699-2b32c5e56fd0","Type":"ContainerStarted","Data":"5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a"}
Jul 31 15:10:12.808493 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:12.808143    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" event={"ID":"8710c59b-e238-4d8f-a699-2b32c5e56fd0","Type":"ContainerStarted","Data":"a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597"}
Jul 31 15:10:12.808493 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:12.808415    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:12.829782 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:12.829711    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podStartSLOduration=2.909463894 podStartE2EDuration="6.829691116s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.920227222s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:10:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:10:12.829417413 +0000 UTC m=+1309.348841478" watchObservedRunningTime="2026-07-31 15:10:12.829691116 +0000 UTC m=+1309.349115154"
Jul 31 15:10:13.813219 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:13.813192    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:19.823825 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:19.823796    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:10:49.825527 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:49.825433    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:10:59.824405 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:10:59.824359    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:11:09.825220 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:09.825179    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:11:19.824588 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:19.824539    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:11:26.995112 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:26.995078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:11:36.808798 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:36.808761    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"]
Jul 31 15:11:36.809269 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:36.809055    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container" containerID="cri-o://a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597" gracePeriod=30
Jul 31 15:11:36.809269 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:36.809078    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kube-rbac-proxy" containerID="cri-o://5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a" gracePeriod=30
Jul 31 15:11:36.957927 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:36.957877    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.32:8080/v2/models/isvc-predictive-xgboost-v2/ready\": dial tcp 10.133.0.32:8080: connect: connection refused"
Jul 31 15:11:37.175516 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:37.175419    2561 generic.go:356] "Generic (PLEG): container finished" podID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerID="5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a" exitCode=2
Jul 31 15:11:37.175516 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:37.175474    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" event={"ID":"8710c59b-e238-4d8f-a699-2b32c5e56fd0","Type":"ContainerDied","Data":"5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a"}
Jul 31 15:11:39.819522 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:39.819473    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.32:8643/healthz\": dial tcp 10.133.0.32:8643: connect: connection refused"
Jul 31 15:11:41.860732 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.860702    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:11:41.955924 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.955832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-rbac-proxy-sar-config\") pod \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") "
Jul 31 15:11:41.955924 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.955897    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8710c59b-e238-4d8f-a699-2b32c5e56fd0-proxy-tls\") pod \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") "
Jul 31 15:11:41.955924 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.955923    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-cabundle-cert\") pod \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") "
Jul 31 15:11:41.956229 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.955950    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kserve-provision-location\") pod \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") "
Jul 31 15:11:41.956229 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.956017    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6sk6b\" (UniqueName: \"kubernetes.io/projected/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-api-access-6sk6b\") pod \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\" (UID: \"8710c59b-e238-4d8f-a699-2b32c5e56fd0\") "
Jul 31 15:11:41.956418 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.956389    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8710c59b-e238-4d8f-a699-2b32c5e56fd0" (UID: "8710c59b-e238-4d8f-a699-2b32c5e56fd0"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:41.956481 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.956411    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8710c59b-e238-4d8f-a699-2b32c5e56fd0" (UID: "8710c59b-e238-4d8f-a699-2b32c5e56fd0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:11:41.956481 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.956424    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8710c59b-e238-4d8f-a699-2b32c5e56fd0" (UID: "8710c59b-e238-4d8f-a699-2b32c5e56fd0"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:11:41.958322 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.958299    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-api-access-6sk6b" (OuterVolumeSpecName: "kube-api-access-6sk6b") pod "8710c59b-e238-4d8f-a699-2b32c5e56fd0" (UID: "8710c59b-e238-4d8f-a699-2b32c5e56fd0"). InnerVolumeSpecName "kube-api-access-6sk6b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:11:41.958549 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.958530    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8710c59b-e238-4d8f-a699-2b32c5e56fd0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8710c59b-e238-4d8f-a699-2b32c5e56fd0" (UID: "8710c59b-e238-4d8f-a699-2b32c5e56fd0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:11:41.984320 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984280    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"]
Jul 31 15:11:41.984722 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984704    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="storage-initializer"
Jul 31 15:11:41.984722 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984723    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="storage-initializer"
Jul 31 15:11:41.985274 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984753    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kube-rbac-proxy"
Jul 31 15:11:41.985274 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984762    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kube-rbac-proxy"
Jul 31 15:11:41.985274 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984778    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container"
Jul 31 15:11:41.985274 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984786    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container"
Jul 31 15:11:41.985274 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984872    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kube-rbac-proxy"
Jul 31 15:11:41.985274 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.984884    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerName="kserve-container"
Jul 31 15:11:41.988310 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.988287    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:41.990715 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.990686    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-predictor-serving-cert\""
Jul 31 15:11:41.990838 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:41.990724    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-predictive-lightgbm-v2-kube-rbac-proxy-sar-config\""
Jul 31 15:11:42.002125 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.002097    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"]
Jul 31 15:11:42.057666 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.057624    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6sk6b\" (UniqueName: \"kubernetes.io/projected/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-api-access-6sk6b\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:11:42.057666 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.057655    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:11:42.057666 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.057667    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8710c59b-e238-4d8f-a699-2b32c5e56fd0-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:11:42.057666 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.057676    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8710c59b-e238-4d8f-a699-2b32c5e56fd0-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:11:42.057941 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.057684    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8710c59b-e238-4d8f-a699-2b32c5e56fd0-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:11:42.158872 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.158831    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldwqg\" (UniqueName: \"kubernetes.io/projected/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-api-access-ldwqg\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.158872 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.158871    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.159092 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.158907    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.159092 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.158965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.159092 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.158988    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.198728 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.198692    2561 generic.go:356] "Generic (PLEG): container finished" podID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" containerID="a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597" exitCode=0
Jul 31 15:11:42.198915 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.198747    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"
Jul 31 15:11:42.198915 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.198775    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" event={"ID":"8710c59b-e238-4d8f-a699-2b32c5e56fd0","Type":"ContainerDied","Data":"a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597"}
Jul 31 15:11:42.198915 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.198843    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb" event={"ID":"8710c59b-e238-4d8f-a699-2b32c5e56fd0","Type":"ContainerDied","Data":"d5d33671c1c27b5ae907e2517919abe95547f7feb07ff300ccffeb60e18f1241"}
Jul 31 15:11:42.198915 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.198872    2561 scope.go:117] "RemoveContainer" containerID="5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a"
Jul 31 15:11:42.207307 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.207282    2561 scope.go:117] "RemoveContainer" containerID="a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597"
Jul 31 15:11:42.215490 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.215433    2561 scope.go:117] "RemoveContainer" containerID="5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3"
Jul 31 15:11:42.217857 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.217831    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"]
Jul 31 15:11:42.222418 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.222395    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-xgboost-v2-predictor-868ff5dd6d-wc4gb"]
Jul 31 15:11:42.225345 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.225327    2561 scope.go:117] "RemoveContainer" containerID="5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a"
Jul 31 15:11:42.225656 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:11:42.225636    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a\": container with ID starting with 5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a not found: ID does not exist" containerID="5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a"
Jul 31 15:11:42.225697 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.225666    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a"} err="failed to get container status \"5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a\": rpc error: code = NotFound desc = could not find container \"5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a\": container with ID starting with 5b5378a9ad4fc634f7787b2b3141bdd23939d3ec40e3a3a806f9fabf92e3170a not found: ID does not exist"
Jul 31 15:11:42.225697 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.225684    2561 scope.go:117] "RemoveContainer" containerID="a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597"
Jul 31 15:11:42.225974 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:11:42.225955    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597\": container with ID starting with a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597 not found: ID does not exist" containerID="a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597"
Jul 31 15:11:42.226031 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.225983    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597"} err="failed to get container status \"a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597\": rpc error: code = NotFound desc = could not find container \"a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597\": container with ID starting with a6aeca360e36b36e4f8237118619f5b877fcdafbe1d6f31a5f2df9dff0a2c597 not found: ID does not exist"
Jul 31 15:11:42.226031 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.226000    2561 scope.go:117] "RemoveContainer" containerID="5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3"
Jul 31 15:11:42.226232 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:11:42.226214    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3\": container with ID starting with 5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3 not found: ID does not exist" containerID="5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3"
Jul 31 15:11:42.226282 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.226235    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3"} err="failed to get container status \"5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3\": rpc error: code = NotFound desc = could not find container \"5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3\": container with ID starting with 5d0c3bf1804ba387974568fe7070e917258a02083e116c01b8d507dc456b53b3 not found: ID does not exist"
Jul 31 15:11:42.259720 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.259673    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.259895 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.259755    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldwqg\" (UniqueName: \"kubernetes.io/projected/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-api-access-ldwqg\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.259895 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.259774    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.259895 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.259807    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.259895 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.259836    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.260247 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.260229    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kserve-provision-location\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.260715 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.260693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-cabundle-cert\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.260762 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.260693    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-rbac-proxy-sar-config\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.262320 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.262304    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-proxy-tls\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.269365 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.269344    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldwqg\" (UniqueName: \"kubernetes.io/projected/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-api-access-ldwqg\") pod \"isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") " pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.299535 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.299503    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:42.444030 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:42.443937    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"]
Jul 31 15:11:43.204543 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:43.204504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" event={"ID":"28c7ab82-2ae1-49ec-b907-8a8e82e488b1","Type":"ContainerStarted","Data":"f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4"}
Jul 31 15:11:43.205043 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:43.204584    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" event={"ID":"28c7ab82-2ae1-49ec-b907-8a8e82e488b1","Type":"ContainerStarted","Data":"cf90ef22a252963dcfc82f32e7214a104e693d0519b76b03456ceba58cbfd878"}
Jul 31 15:11:43.961825 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:43.961788    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8710c59b-e238-4d8f-a699-2b32c5e56fd0" path="/var/lib/kubelet/pods/8710c59b-e238-4d8f-a699-2b32c5e56fd0/volumes"
Jul 31 15:11:47.224944 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:47.224903    2561 generic.go:356] "Generic (PLEG): container finished" podID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerID="f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4" exitCode=0
Jul 31 15:11:47.225343 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:47.224975    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" event={"ID":"28c7ab82-2ae1-49ec-b907-8a8e82e488b1","Type":"ContainerDied","Data":"f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4"}
Jul 31 15:11:48.232188 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:48.232135    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" event={"ID":"28c7ab82-2ae1-49ec-b907-8a8e82e488b1","Type":"ContainerStarted","Data":"017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065"}
Jul 31 15:11:48.232635 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:48.232216    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" event={"ID":"28c7ab82-2ae1-49ec-b907-8a8e82e488b1","Type":"ContainerStarted","Data":"3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16"}
Jul 31 15:11:48.232635 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:48.232433    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:48.232635 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:48.232568    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:11:48.252986 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:48.252918    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" podStartSLOduration=3.221782973 podStartE2EDuration="7.252899275s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.031116302s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:11:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:11:48.251450358 +0000 UTC m=+1404.770874398" watchObservedRunningTime="2026-07-31 15:11:48.252899275 +0000 UTC m=+1404.772323313"
Jul 31 15:11:54.243342 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:11:54.243317    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:12:24.244480 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:12:24.244442    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 31 15:12:34.244295 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:12:34.244254    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 31 15:12:44.244464 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:12:44.244427    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 31 15:12:54.244681 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:12:54.244642    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kserve-container" probeResult="failure" output="Get \"http://10.133.0.33:8080/v2/models/isvc-predictive-lightgbm-v2/ready\": dial tcp 10.133.0.33:8080: connect: connection refused"
Jul 31 15:13:01.994120 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:01.994084    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:13:12.120194 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:12.120146    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"]
Jul 31 15:13:12.120648 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:12.120423    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kserve-container" containerID="cri-o://3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16" gracePeriod=30
Jul 31 15:13:12.120648 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:12.120479    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kube-rbac-proxy" containerID="cri-o://017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065" gracePeriod=30
Jul 31 15:13:12.603464 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:12.603434    2561 generic.go:356] "Generic (PLEG): container finished" podID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerID="017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065" exitCode=2
Jul 31 15:13:12.603636 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:12.603504    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" event={"ID":"28c7ab82-2ae1-49ec-b907-8a8e82e488b1","Type":"ContainerDied","Data":"017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065"}
Jul 31 15:13:14.238374 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:14.238334    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.33:8643/healthz\": dial tcp 10.133.0.33:8643: connect: connection refused"
Jul 31 15:13:17.466242 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.466208    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:13:17.601501 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.601399    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-rbac-proxy-sar-config\") pod \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") "
Jul 31 15:13:17.601501 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.601455    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ldwqg\" (UniqueName: \"kubernetes.io/projected/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-api-access-ldwqg\") pod \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") "
Jul 31 15:13:17.601763 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.601510    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-cabundle-cert\") pod \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") "
Jul 31 15:13:17.601763 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.601530    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-proxy-tls\") pod \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") "
Jul 31 15:13:17.601763 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.601549    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kserve-provision-location\") pod \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\" (UID: \"28c7ab82-2ae1-49ec-b907-8a8e82e488b1\") "
Jul 31 15:13:17.601944 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.601864    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "28c7ab82-2ae1-49ec-b907-8a8e82e488b1" (UID: "28c7ab82-2ae1-49ec-b907-8a8e82e488b1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:13:17.601944 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.601874    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "28c7ab82-2ae1-49ec-b907-8a8e82e488b1" (UID: "28c7ab82-2ae1-49ec-b907-8a8e82e488b1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:13:17.601944 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.601897    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "28c7ab82-2ae1-49ec-b907-8a8e82e488b1" (UID: "28c7ab82-2ae1-49ec-b907-8a8e82e488b1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:13:17.603735 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.603708    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "28c7ab82-2ae1-49ec-b907-8a8e82e488b1" (UID: "28c7ab82-2ae1-49ec-b907-8a8e82e488b1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:13:17.603829 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.603751    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-api-access-ldwqg" (OuterVolumeSpecName: "kube-api-access-ldwqg") pod "28c7ab82-2ae1-49ec-b907-8a8e82e488b1" (UID: "28c7ab82-2ae1-49ec-b907-8a8e82e488b1"). InnerVolumeSpecName "kube-api-access-ldwqg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:13:17.626150 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.626113    2561 generic.go:356] "Generic (PLEG): container finished" podID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerID="3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16" exitCode=0
Jul 31 15:13:17.626320 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.626206    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"
Jul 31 15:13:17.626320 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.626206    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" event={"ID":"28c7ab82-2ae1-49ec-b907-8a8e82e488b1","Type":"ContainerDied","Data":"3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16"}
Jul 31 15:13:17.626320 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.626282    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j" event={"ID":"28c7ab82-2ae1-49ec-b907-8a8e82e488b1","Type":"ContainerDied","Data":"cf90ef22a252963dcfc82f32e7214a104e693d0519b76b03456ceba58cbfd878"}
Jul 31 15:13:17.626427 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.626324    2561 scope.go:117] "RemoveContainer" containerID="017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065"
Jul 31 15:13:17.635561 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.635542    2561 scope.go:117] "RemoveContainer" containerID="3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16"
Jul 31 15:13:17.643830 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.643806    2561 scope.go:117] "RemoveContainer" containerID="f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4"
Jul 31 15:13:17.650046 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.650023    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"]
Jul 31 15:13:17.652371 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.652353    2561 scope.go:117] "RemoveContainer" containerID="017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065"
Jul 31 15:13:17.652677 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:13:17.652653    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065\": container with ID starting with 017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065 not found: ID does not exist" containerID="017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065"
Jul 31 15:13:17.652806 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.652691    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065"} err="failed to get container status \"017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065\": rpc error: code = NotFound desc = could not find container \"017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065\": container with ID starting with 017814aa70e7bf1afc730a52e9359369f223fe804cf7d9d0b52d50a4be265065 not found: ID does not exist"
Jul 31 15:13:17.652806 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.652716    2561 scope.go:117] "RemoveContainer" containerID="3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16"
Jul 31 15:13:17.653006 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:13:17.652989    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16\": container with ID starting with 3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16 not found: ID does not exist" containerID="3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16"
Jul 31 15:13:17.653077 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.653013    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16"} err="failed to get container status \"3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16\": rpc error: code = NotFound desc = could not find container \"3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16\": container with ID starting with 3231caa28ef3f388abe88d7f5be29375b6b7780ebbf8c5f094bb88e38a8f0c16 not found: ID does not exist"
Jul 31 15:13:17.653077 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.653032    2561 scope.go:117] "RemoveContainer" containerID="f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4"
Jul 31 15:13:17.653529 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:13:17.653477    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4\": container with ID starting with f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4 not found: ID does not exist" containerID="f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4"
Jul 31 15:13:17.653529 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.653506    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4"} err="failed to get container status \"f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4\": rpc error: code = NotFound desc = could not find container \"f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4\": container with ID starting with f3e1c70670b7a1f9afdd5a9c81deb936ea5ff73f00f193868d8a7ee36503dbe4 not found: ID does not exist"
Jul 31 15:13:17.655539 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.655519    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-predictive-lightgbm-v2-predictor-545b7cc587-6cf4j"]
Jul 31 15:13:17.703217 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.703183    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldwqg\" (UniqueName: \"kubernetes.io/projected/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-api-access-ldwqg\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:13:17.703217 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.703212    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:13:17.703217 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.703224    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:13:17.703428 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.703232    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:13:17.703428 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.703241    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/28c7ab82-2ae1-49ec-b907-8a8e82e488b1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:13:17.961286 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:17.961202    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" path="/var/lib/kubelet/pods/28c7ab82-2ae1-49ec-b907-8a8e82e488b1/volumes"
Jul 31 15:13:23.999023 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:23.998994    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:13:24.002472 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:13:24.002449    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:14:35.190051 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190015    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"]
Jul 31 15:14:35.192299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190392    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="storage-initializer"
Jul 31 15:14:35.192299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190405    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="storage-initializer"
Jul 31 15:14:35.192299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190424    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kube-rbac-proxy"
Jul 31 15:14:35.192299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190432    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kube-rbac-proxy"
Jul 31 15:14:35.192299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190442    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kserve-container"
Jul 31 15:14:35.192299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190448    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kserve-container"
Jul 31 15:14:35.192299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190512    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kube-rbac-proxy"
Jul 31 15:14:35.192299 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.190520    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="28c7ab82-2ae1-49ec-b907-8a8e82e488b1" containerName="kserve-container"
Jul 31 15:14:35.193414 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.193399    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.195959 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.195935    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:14:35.196085 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.196024    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:14:35.196178 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.196089    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-kube-rbac-proxy-sar-config\""
Jul 31 15:14:35.196245 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.196176    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:14:35.196304 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.196249    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:14:35.196688 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.196672    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-autogluon-ts-v1-noslash-predictor-serving-cert\""
Jul 31 15:14:35.196735 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.196723    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:14:35.204958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.204927    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"]
Jul 31 15:14:35.285429 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.285395    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.285429 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.285436    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f715041-6295-489f-bac5-29e1405c2d20-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.285663 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.285455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.285663 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.285483    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f715041-6295-489f-bac5-29e1405c2d20-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.285663 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.285541    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghbw8\" (UniqueName: \"kubernetes.io/projected/0f715041-6295-489f-bac5-29e1405c2d20-kube-api-access-ghbw8\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.386102 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.386068    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghbw8\" (UniqueName: \"kubernetes.io/projected/0f715041-6295-489f-bac5-29e1405c2d20-kube-api-access-ghbw8\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.386354 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.386137    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.386354 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.386186    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f715041-6295-489f-bac5-29e1405c2d20-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.386354 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.386202    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.386354 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.386240    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f715041-6295-489f-bac5-29e1405c2d20-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.386696 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.386673    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f715041-6295-489f-bac5-29e1405c2d20-kserve-provision-location\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.386877 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.386858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-kube-rbac-proxy-sar-config\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.386935 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.386878    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-cabundle-cert\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.388717 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.388696    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f715041-6295-489f-bac5-29e1405c2d20-proxy-tls\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.393910 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.393888    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghbw8\" (UniqueName: \"kubernetes.io/projected/0f715041-6295-489f-bac5-29e1405c2d20-kube-api-access-ghbw8\") pod \"isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") " pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.505449 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.505413    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:14:35.641516 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.641489    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"]
Jul 31 15:14:35.644140 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:14:35.644107    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f715041_6295_489f_bac5_29e1405c2d20.slice/crio-7d4c0a0fd5616aea309d9e20b30862e60daab7702391fcdd4be354df2bb9359b WatchSource:0}: Error finding container 7d4c0a0fd5616aea309d9e20b30862e60daab7702391fcdd4be354df2bb9359b: Status 404 returned error can't find the container with id 7d4c0a0fd5616aea309d9e20b30862e60daab7702391fcdd4be354df2bb9359b
Jul 31 15:14:35.950635 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.950539    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" event={"ID":"0f715041-6295-489f-bac5-29e1405c2d20","Type":"ContainerStarted","Data":"d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57"}
Jul 31 15:14:35.950635 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:35.950596    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" event={"ID":"0f715041-6295-489f-bac5-29e1405c2d20","Type":"ContainerStarted","Data":"7d4c0a0fd5616aea309d9e20b30862e60daab7702391fcdd4be354df2bb9359b"}
Jul 31 15:14:44.987814 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:44.987778    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f715041-6295-489f-bac5-29e1405c2d20" containerID="d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57" exitCode=0
Jul 31 15:14:44.988266 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:44.987852    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" event={"ID":"0f715041-6295-489f-bac5-29e1405c2d20","Type":"ContainerDied","Data":"d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57"}
Jul 31 15:14:44.989256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:14:44.989240    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:16:50.608500 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:50.608452    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" event={"ID":"0f715041-6295-489f-bac5-29e1405c2d20","Type":"ContainerStarted","Data":"02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795"}
Jul 31 15:16:50.609223 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:50.608579    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:16:50.609223 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:50.608617    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" event={"ID":"0f715041-6295-489f-bac5-29e1405c2d20","Type":"ContainerStarted","Data":"78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d"}
Jul 31 15:16:50.636482 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:50.636419    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" podStartSLOduration=1.361807425 podStartE2EDuration="2m15.636396434s" totalImagesPullingTime="2m5.117240958s" totalInitContainerRuntime="9.157348051s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:14:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:16:50.635693694 +0000 UTC m=+1707.155117733" watchObservedRunningTime="2026-07-31 15:16:50.636396434 +0000 UTC m=+1707.155820473"
Jul 31 15:16:51.613396 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:51.613354    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:16:51.615046 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:51.615016    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:16:52.618737 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:52.618697    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:16:57.623280 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:57.623248    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:16:57.623909 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:16:57.623881    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.34:8080: connect: connection refused"
Jul 31 15:17:07.625222 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:07.625188    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:17:20.781325 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:20.781291    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"]
Jul 31 15:17:20.781860 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:20.781519    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kserve-container" containerID="cri-o://78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d" gracePeriod=30
Jul 31 15:17:20.781860 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:20.781576    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kube-rbac-proxy" containerID="cri-o://02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795" gracePeriod=30
Jul 31 15:17:21.749761 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:21.749723    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f715041-6295-489f-bac5-29e1405c2d20" containerID="02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795" exitCode=2
Jul 31 15:17:21.749957 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:21.749799    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" event={"ID":"0f715041-6295-489f-bac5-29e1405c2d20","Type":"ContainerDied","Data":"02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795"}
Jul 31 15:17:22.250814 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.250781    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:17:22.360910 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.360811    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-cabundle-cert\") pod \"0f715041-6295-489f-bac5-29e1405c2d20\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") "
Jul 31 15:17:22.360910 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.360904    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f715041-6295-489f-bac5-29e1405c2d20-kserve-provision-location\") pod \"0f715041-6295-489f-bac5-29e1405c2d20\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") "
Jul 31 15:17:22.361147 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.360936    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f715041-6295-489f-bac5-29e1405c2d20-proxy-tls\") pod \"0f715041-6295-489f-bac5-29e1405c2d20\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") "
Jul 31 15:17:22.361147 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.360987    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ghbw8\" (UniqueName: \"kubernetes.io/projected/0f715041-6295-489f-bac5-29e1405c2d20-kube-api-access-ghbw8\") pod \"0f715041-6295-489f-bac5-29e1405c2d20\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") "
Jul 31 15:17:22.361147 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.361023    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-kube-rbac-proxy-sar-config\") pod \"0f715041-6295-489f-bac5-29e1405c2d20\" (UID: \"0f715041-6295-489f-bac5-29e1405c2d20\") "
Jul 31 15:17:22.361303 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.361197    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "0f715041-6295-489f-bac5-29e1405c2d20" (UID: "0f715041-6295-489f-bac5-29e1405c2d20"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:22.361303 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.361273    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:17:22.361481 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.361455    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "0f715041-6295-489f-bac5-29e1405c2d20" (UID: "0f715041-6295-489f-bac5-29e1405c2d20"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:17:22.362018 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.361998    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f715041-6295-489f-bac5-29e1405c2d20-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "0f715041-6295-489f-bac5-29e1405c2d20" (UID: "0f715041-6295-489f-bac5-29e1405c2d20"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:17:22.363232 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.363204    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f715041-6295-489f-bac5-29e1405c2d20-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "0f715041-6295-489f-bac5-29e1405c2d20" (UID: "0f715041-6295-489f-bac5-29e1405c2d20"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:17:22.363232 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.363204    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f715041-6295-489f-bac5-29e1405c2d20-kube-api-access-ghbw8" (OuterVolumeSpecName: "kube-api-access-ghbw8") pod "0f715041-6295-489f-bac5-29e1405c2d20" (UID: "0f715041-6295-489f-bac5-29e1405c2d20"). InnerVolumeSpecName "kube-api-access-ghbw8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:17:22.462294 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.462252    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghbw8\" (UniqueName: \"kubernetes.io/projected/0f715041-6295-489f-bac5-29e1405c2d20-kube-api-access-ghbw8\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:17:22.462294 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.462288    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/0f715041-6295-489f-bac5-29e1405c2d20-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:17:22.462294 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.462299    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/0f715041-6295-489f-bac5-29e1405c2d20-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:17:22.462540 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.462309    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/0f715041-6295-489f-bac5-29e1405c2d20-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:17:22.756332 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.756298    2561 generic.go:356] "Generic (PLEG): container finished" podID="0f715041-6295-489f-bac5-29e1405c2d20" containerID="78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d" exitCode=0
Jul 31 15:17:22.756523 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.756340    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" event={"ID":"0f715041-6295-489f-bac5-29e1405c2d20","Type":"ContainerDied","Data":"78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d"}
Jul 31 15:17:22.756523 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.756357    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"
Jul 31 15:17:22.756523 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.756389    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb" event={"ID":"0f715041-6295-489f-bac5-29e1405c2d20","Type":"ContainerDied","Data":"7d4c0a0fd5616aea309d9e20b30862e60daab7702391fcdd4be354df2bb9359b"}
Jul 31 15:17:22.756523 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.756416    2561 scope.go:117] "RemoveContainer" containerID="02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795"
Jul 31 15:17:22.765576 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.765558    2561 scope.go:117] "RemoveContainer" containerID="78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d"
Jul 31 15:17:22.774222 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.774191    2561 scope.go:117] "RemoveContainer" containerID="d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57"
Jul 31 15:17:22.779406 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.779374    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"]
Jul 31 15:17:22.782918 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.782896    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-autogluon-ts-v1-noslash-predictor-b54fbf585-z6vgb"]
Jul 31 15:17:22.782918 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.782912    2561 scope.go:117] "RemoveContainer" containerID="02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795"
Jul 31 15:17:22.783350 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:17:22.783331    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795\": container with ID starting with 02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795 not found: ID does not exist" containerID="02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795"
Jul 31 15:17:22.783422 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.783363    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795"} err="failed to get container status \"02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795\": rpc error: code = NotFound desc = could not find container \"02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795\": container with ID starting with 02e253c30cef8117c2627b8776a8daff62d5273fac018237c1bde73c066b0795 not found: ID does not exist"
Jul 31 15:17:22.783422 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.783383    2561 scope.go:117] "RemoveContainer" containerID="78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d"
Jul 31 15:17:22.783676 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:17:22.783659    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d\": container with ID starting with 78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d not found: ID does not exist" containerID="78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d"
Jul 31 15:17:22.783743 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.783686    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d"} err="failed to get container status \"78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d\": rpc error: code = NotFound desc = could not find container \"78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d\": container with ID starting with 78d10e116df029a3a33724191d5306907983c9665f737fea5711b9d341c00f6d not found: ID does not exist"
Jul 31 15:17:22.783743 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.783710    2561 scope.go:117] "RemoveContainer" containerID="d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57"
Jul 31 15:17:22.783960 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:17:22.783944    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57\": container with ID starting with d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57 not found: ID does not exist" containerID="d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57"
Jul 31 15:17:22.784002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:22.783967    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57"} err="failed to get container status \"d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57\": rpc error: code = NotFound desc = could not find container \"d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57\": container with ID starting with d8cc2aeb9be8ecfef66d7f7cbea9cfccaeb0bcbaa68f22acbe69858dae900e57 not found: ID does not exist"
Jul 31 15:17:23.960847 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:17:23.960817    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f715041-6295-489f-bac5-29e1405c2d20" path="/var/lib/kubelet/pods/0f715041-6295-489f-bac5-29e1405c2d20/volumes"
Jul 31 15:18:24.026431 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:18:24.026400    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:18:24.030743 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:18:24.030722    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:20:41.281645 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.281608    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"]
Jul 31 15:20:41.282094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.281925    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kube-rbac-proxy"
Jul 31 15:20:41.282094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.281936    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kube-rbac-proxy"
Jul 31 15:20:41.282094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.281948    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="storage-initializer"
Jul 31 15:20:41.282094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.281954    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="storage-initializer"
Jul 31 15:20:41.282094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.281970    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kserve-container"
Jul 31 15:20:41.282094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.281976    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kserve-container"
Jul 31 15:20:41.282094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.282028    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kube-rbac-proxy"
Jul 31 15:20:41.282094 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.282035    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f715041-6295-489f-bac5-29e1405c2d20" containerName="kserve-container"
Jul 31 15:20:41.284234 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.284217    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.286578 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.286556    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:20:41.286723 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.286560    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:20:41.286844 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.286828    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-kube-rbac-proxy-sar-config\""
Jul 31 15:20:41.287027 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.287013    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:20:41.287112 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.287094    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:20:41.287323 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.287307    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-promote-v2-predictor-serving-cert\""
Jul 31 15:20:41.287409 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.287395    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:20:41.295857 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.295832    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"]
Jul 31 15:20:41.409452 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.409416    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3604a4c-400d-4463-8286-21f06a0067a1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.409636 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.409459    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.409636 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.409519    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.409636 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.409597    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58g5n\" (UniqueName: \"kubernetes.io/projected/b3604a4c-400d-4463-8286-21f06a0067a1-kube-api-access-58g5n\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.409773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.409666    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3604a4c-400d-4463-8286-21f06a0067a1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.510574 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.510534    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58g5n\" (UniqueName: \"kubernetes.io/projected/b3604a4c-400d-4463-8286-21f06a0067a1-kube-api-access-58g5n\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.510749 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.510604    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3604a4c-400d-4463-8286-21f06a0067a1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.510749 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.510646    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3604a4c-400d-4463-8286-21f06a0067a1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.510749 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.510672    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.510749 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.510698    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.511185 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.511134    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3604a4c-400d-4463-8286-21f06a0067a1-kserve-provision-location\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.511428 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.511408    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-cabundle-cert\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.511504 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.511450    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-kube-rbac-proxy-sar-config\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.513184 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.513148    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3604a4c-400d-4463-8286-21f06a0067a1-proxy-tls\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.518800 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.518780    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58g5n\" (UniqueName: \"kubernetes.io/projected/b3604a4c-400d-4463-8286-21f06a0067a1-kube-api-access-58g5n\") pod \"isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") " pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.595322 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.595238    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:41.728587 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:41.728505    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"]
Jul 31 15:20:41.731446 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:20:41.731419    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3604a4c_400d_4463_8286_21f06a0067a1.slice/crio-0e2f070f4310ab7b87502d735ce8a9540dc62d8d06e87fb686f0660abe80f466 WatchSource:0}: Error finding container 0e2f070f4310ab7b87502d735ce8a9540dc62d8d06e87fb686f0660abe80f466: Status 404 returned error can't find the container with id 0e2f070f4310ab7b87502d735ce8a9540dc62d8d06e87fb686f0660abe80f466
Jul 31 15:20:42.581486 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:42.581446    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" event={"ID":"b3604a4c-400d-4463-8286-21f06a0067a1","Type":"ContainerStarted","Data":"572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e"}
Jul 31 15:20:42.581907 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:42.581506    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" event={"ID":"b3604a4c-400d-4463-8286-21f06a0067a1","Type":"ContainerStarted","Data":"0e2f070f4310ab7b87502d735ce8a9540dc62d8d06e87fb686f0660abe80f466"}
Jul 31 15:20:45.595351 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:45.595316    2561 generic.go:356] "Generic (PLEG): container finished" podID="b3604a4c-400d-4463-8286-21f06a0067a1" containerID="572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e" exitCode=0
Jul 31 15:20:45.595726 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:45.595388    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" event={"ID":"b3604a4c-400d-4463-8286-21f06a0067a1","Type":"ContainerDied","Data":"572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e"}
Jul 31 15:20:45.596511 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:45.596496    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:20:59.675776 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:59.675739    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" event={"ID":"b3604a4c-400d-4463-8286-21f06a0067a1","Type":"ContainerStarted","Data":"cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d"}
Jul 31 15:20:59.676240 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:59.675803    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" event={"ID":"b3604a4c-400d-4463-8286-21f06a0067a1","Type":"ContainerStarted","Data":"8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9"}
Jul 31 15:20:59.676240 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:59.676113    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:59.676240 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:59.676190    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:20:59.677699 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:59.677671    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:20:59.696683 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:20:59.696626    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podStartSLOduration=1.827447254 podStartE2EDuration="18.696608752s" totalImagesPullingTime="13.194689278s" totalInitContainerRuntime="3.67447222s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:20:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:20:59.694995148 +0000 UTC m=+1956.214419186" watchObservedRunningTime="2026-07-31 15:20:59.696608752 +0000 UTC m=+1956.216032790"
Jul 31 15:21:00.682095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:21:00.682051    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:21:05.687528 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:21:05.687496    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:21:05.688007 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:21:05.687981    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:21:15.688268 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:21:15.688185    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:21:25.688402 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:21:25.688361    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:21:35.688002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:21:35.687956    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:21:45.687975 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:21:45.687933    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:21:55.688717 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:21:55.688665    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:22:05.688039 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:05.687994    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:22:15.689422 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:15.689391    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:22:21.467582 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:21.467551    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"]
Jul 31 15:22:21.467994 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:21.467841    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" containerID="cri-o://8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9" gracePeriod=30
Jul 31 15:22:21.467994 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:21.467863    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kube-rbac-proxy" containerID="cri-o://cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d" gracePeriod=30
Jul 31 15:22:22.025180 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:22.025137    2561 generic.go:356] "Generic (PLEG): container finished" podID="b3604a4c-400d-4463-8286-21f06a0067a1" containerID="cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d" exitCode=2
Jul 31 15:22:22.025364 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:22.025180    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" event={"ID":"b3604a4c-400d-4463-8286-21f06a0067a1","Type":"ContainerDied","Data":"cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d"}
Jul 31 15:22:25.682909 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:25.682872    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.35:8643/healthz\": dial tcp 10.133.0.35:8643: connect: connection refused"
Jul 31 15:22:25.688246 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:25.688217    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.35:8080: connect: connection refused"
Jul 31 15:22:26.415063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.415038    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:22:26.454774 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.454684    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-cabundle-cert\") pod \"b3604a4c-400d-4463-8286-21f06a0067a1\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") "
Jul 31 15:22:26.454774 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.454732    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3604a4c-400d-4463-8286-21f06a0067a1-kserve-provision-location\") pod \"b3604a4c-400d-4463-8286-21f06a0067a1\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") "
Jul 31 15:22:26.455024 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.454868    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-kube-rbac-proxy-sar-config\") pod \"b3604a4c-400d-4463-8286-21f06a0067a1\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") "
Jul 31 15:22:26.455024 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.454912    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3604a4c-400d-4463-8286-21f06a0067a1-proxy-tls\") pod \"b3604a4c-400d-4463-8286-21f06a0067a1\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") "
Jul 31 15:22:26.455024 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.454964    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58g5n\" (UniqueName: \"kubernetes.io/projected/b3604a4c-400d-4463-8286-21f06a0067a1-kube-api-access-58g5n\") pod \"b3604a4c-400d-4463-8286-21f06a0067a1\" (UID: \"b3604a4c-400d-4463-8286-21f06a0067a1\") "
Jul 31 15:22:26.455192 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.455068    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b3604a4c-400d-4463-8286-21f06a0067a1-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "b3604a4c-400d-4463-8286-21f06a0067a1" (UID: "b3604a4c-400d-4463-8286-21f06a0067a1"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:22:26.455241 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.455190    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "b3604a4c-400d-4463-8286-21f06a0067a1" (UID: "b3604a4c-400d-4463-8286-21f06a0067a1"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:22:26.455241 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.455227    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b3604a4c-400d-4463-8286-21f06a0067a1" (UID: "b3604a4c-400d-4463-8286-21f06a0067a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:22:26.455369 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.455350    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.455436 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.455373    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/b3604a4c-400d-4463-8286-21f06a0067a1-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.455436 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.455387    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/b3604a4c-400d-4463-8286-21f06a0067a1-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.457221 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.457199    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b3604a4c-400d-4463-8286-21f06a0067a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b3604a4c-400d-4463-8286-21f06a0067a1" (UID: "b3604a4c-400d-4463-8286-21f06a0067a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:22:26.457510 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.457482    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3604a4c-400d-4463-8286-21f06a0067a1-kube-api-access-58g5n" (OuterVolumeSpecName: "kube-api-access-58g5n") pod "b3604a4c-400d-4463-8286-21f06a0067a1" (UID: "b3604a4c-400d-4463-8286-21f06a0067a1"). InnerVolumeSpecName "kube-api-access-58g5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:22:26.555761 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.555726    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b3604a4c-400d-4463-8286-21f06a0067a1-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.555761 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.555757    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58g5n\" (UniqueName: \"kubernetes.io/projected/b3604a4c-400d-4463-8286-21f06a0067a1-kube-api-access-58g5n\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:22:26.678380 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678345    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"]
Jul 31 15:22:26.678772 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678753    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kube-rbac-proxy"
Jul 31 15:22:26.678867 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678777    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kube-rbac-proxy"
Jul 31 15:22:26.678867 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678809    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container"
Jul 31 15:22:26.678867 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678819    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container"
Jul 31 15:22:26.678867 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678839    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="storage-initializer"
Jul 31 15:22:26.678867 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678849    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="storage-initializer"
Jul 31 15:22:26.679143 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678932    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kube-rbac-proxy"
Jul 31 15:22:26.679143 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.678946    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" containerName="kserve-container"
Jul 31 15:22:26.681245 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.681224    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.685335 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.685316    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-predictor-serving-cert\""
Jul 31 15:22:26.687167 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.687135    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-rollback-kube-rbac-proxy-sar-config\""
Jul 31 15:22:26.700978 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.700950    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"]
Jul 31 15:22:26.757970 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.757936    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kgcd6\" (UniqueName: \"kubernetes.io/projected/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-api-access-kgcd6\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.758148 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.757990    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.758148 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.758079    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.758148 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.758114    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.758148 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.758145    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.859042 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.859001    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.859042 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.859045    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.859355 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.859095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kgcd6\" (UniqueName: \"kubernetes.io/projected/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-api-access-kgcd6\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.859355 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.859123    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.859355 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.859185    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.859355 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:22:26.859245    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-rollback-predictor-serving-cert: secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 31 15:22:26.859355 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:22:26.859357    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls podName:8f2a4620-4da0-4fc8-8745-fc13f77402d7 nodeName:}" failed. No retries permitted until 2026-07-31 15:22:27.359333443 +0000 UTC m=+2043.878757473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls") pod "isvc-canary-rollback-predictor-7f985c86-6w4dd" (UID: "8f2a4620-4da0-4fc8-8745-fc13f77402d7") : secret "isvc-canary-rollback-predictor-serving-cert" not found
Jul 31 15:22:26.859698 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.859617    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kserve-provision-location\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.859952 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.859931    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-rbac-proxy-sar-config\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.860030 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.859970    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-cabundle-cert\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:26.878085 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:26.878058    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kgcd6\" (UniqueName: \"kubernetes.io/projected/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-api-access-kgcd6\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:27.048880 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.048798    2561 generic.go:356] "Generic (PLEG): container finished" podID="b3604a4c-400d-4463-8286-21f06a0067a1" containerID="8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9" exitCode=0
Jul 31 15:22:27.048880 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.048860    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"
Jul 31 15:22:27.049069 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.048865    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" event={"ID":"b3604a4c-400d-4463-8286-21f06a0067a1","Type":"ContainerDied","Data":"8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9"}
Jul 31 15:22:27.049069 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.048988    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2" event={"ID":"b3604a4c-400d-4463-8286-21f06a0067a1","Type":"ContainerDied","Data":"0e2f070f4310ab7b87502d735ce8a9540dc62d8d06e87fb686f0660abe80f466"}
Jul 31 15:22:27.049069 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.049020    2561 scope.go:117] "RemoveContainer" containerID="cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d"
Jul 31 15:22:27.057408 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.057387    2561 scope.go:117] "RemoveContainer" containerID="8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9"
Jul 31 15:22:27.065031 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.065013    2561 scope.go:117] "RemoveContainer" containerID="572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e"
Jul 31 15:22:27.072640 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.072624    2561 scope.go:117] "RemoveContainer" containerID="cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d"
Jul 31 15:22:27.072894 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:22:27.072877    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d\": container with ID starting with cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d not found: ID does not exist" containerID="cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d"
Jul 31 15:22:27.072944 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.072904    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d"} err="failed to get container status \"cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d\": rpc error: code = NotFound desc = could not find container \"cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d\": container with ID starting with cf84c92b56cec55c9d5d5c26db9859ebc4dc695b487fbb9125053423e2788e0d not found: ID does not exist"
Jul 31 15:22:27.072944 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.072931    2561 scope.go:117] "RemoveContainer" containerID="8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9"
Jul 31 15:22:27.073257 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:22:27.073232    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9\": container with ID starting with 8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9 not found: ID does not exist" containerID="8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9"
Jul 31 15:22:27.073317 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.073264    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9"} err="failed to get container status \"8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9\": rpc error: code = NotFound desc = could not find container \"8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9\": container with ID starting with 8e1c5b980429bdba99579c3c854ff1b1d505227d09780027c7478c9ef46801d9 not found: ID does not exist"
Jul 31 15:22:27.073317 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.073282    2561 scope.go:117] "RemoveContainer" containerID="572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e"
Jul 31 15:22:27.073528 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:22:27.073514    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e\": container with ID starting with 572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e not found: ID does not exist" containerID="572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e"
Jul 31 15:22:27.073569 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.073532    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e"} err="failed to get container status \"572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e\": rpc error: code = NotFound desc = could not find container \"572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e\": container with ID starting with 572588a45edaac8174d46edd42543cea1acd0599b887a508681a78e870e53c8e not found: ID does not exist"
Jul 31 15:22:27.089356 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.089327    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"]
Jul 31 15:22:27.101606 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.101579    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-promote-v2-predictor-b87c95ccc-7v4q2"]
Jul 31 15:22:27.363673 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.363588    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:27.366010 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.365977    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls\") pod \"isvc-canary-rollback-predictor-7f985c86-6w4dd\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") " pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:27.591825 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.591781    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:27.729469 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.729446    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"]
Jul 31 15:22:27.731554 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:22:27.731519    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f2a4620_4da0_4fc8_8745_fc13f77402d7.slice/crio-023bab83cfbfe19ab485bb5780b8b81b90a97aa9684d58a53b1b9915264cee90 WatchSource:0}: Error finding container 023bab83cfbfe19ab485bb5780b8b81b90a97aa9684d58a53b1b9915264cee90: Status 404 returned error can't find the container with id 023bab83cfbfe19ab485bb5780b8b81b90a97aa9684d58a53b1b9915264cee90
Jul 31 15:22:27.962526 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:27.962439    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b3604a4c-400d-4463-8286-21f06a0067a1" path="/var/lib/kubelet/pods/b3604a4c-400d-4463-8286-21f06a0067a1/volumes"
Jul 31 15:22:28.056830 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:28.056794    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" event={"ID":"8f2a4620-4da0-4fc8-8745-fc13f77402d7","Type":"ContainerStarted","Data":"309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b"}
Jul 31 15:22:28.056977 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:28.056860    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" event={"ID":"8f2a4620-4da0-4fc8-8745-fc13f77402d7","Type":"ContainerStarted","Data":"023bab83cfbfe19ab485bb5780b8b81b90a97aa9684d58a53b1b9915264cee90"}
Jul 31 15:22:32.077283 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:32.077250    2561 generic.go:356] "Generic (PLEG): container finished" podID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerID="309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b" exitCode=0
Jul 31 15:22:32.077659 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:32.077327    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" event={"ID":"8f2a4620-4da0-4fc8-8745-fc13f77402d7","Type":"ContainerDied","Data":"309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b"}
Jul 31 15:22:33.087638 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:33.087602    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" event={"ID":"8f2a4620-4da0-4fc8-8745-fc13f77402d7","Type":"ContainerStarted","Data":"81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5"}
Jul 31 15:22:33.088044 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:33.087668    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" event={"ID":"8f2a4620-4da0-4fc8-8745-fc13f77402d7","Type":"ContainerStarted","Data":"a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4"}
Jul 31 15:22:33.088044 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:33.087859    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:33.113369 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:33.113316    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podStartSLOduration=3.425688096 podStartE2EDuration="7.11329989s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.687611794s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:22:33.111514821 +0000 UTC m=+2049.630938872" watchObservedRunningTime="2026-07-31 15:22:33.11329989 +0000 UTC m=+2049.632723927"
Jul 31 15:22:34.092654 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:34.092620    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:34.094075 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:34.094046    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:22:35.097168 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:35.097125    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:22:40.102383 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:40.102354    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:22:40.102879 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:40.102807    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:22:50.103093 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:22:50.102999    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:23:00.103331 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:23:00.103295    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:23:10.102849 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:23:10.102810    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:23:20.103067 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:23:20.103025    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:23:24.055954 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:23:24.055925    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:23:24.060807 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:23:24.060782    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:23:30.103774 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:23:30.103727    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:23:40.103890 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:23:40.103847    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:23:50.104047 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:23:50.104017    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:25:46.790647 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:46.790551    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"]
Jul 31 15:25:46.791357 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:46.790862    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" containerID="cri-o://a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4" gracePeriod=30
Jul 31 15:25:46.791357 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:46.790909    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kube-rbac-proxy" containerID="cri-o://81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5" gracePeriod=30
Jul 31 15:25:46.923889 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:46.923859    2561 generic.go:356] "Generic (PLEG): container finished" podID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerID="81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5" exitCode=2
Jul 31 15:25:46.924058 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:46.923902    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" event={"ID":"8f2a4620-4da0-4fc8-8745-fc13f77402d7","Type":"ContainerDied","Data":"81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5"}
Jul 31 15:25:50.097710 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:50.097657    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.36:8643/healthz\": dial tcp 10.133.0.36:8643: connect: connection refused"
Jul 31 15:25:50.103150 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:50.103122    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.36:8080: connect: connection refused"
Jul 31 15:25:51.640028 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.639999    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:25:51.727904 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.727878    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kgcd6\" (UniqueName: \"kubernetes.io/projected/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-api-access-kgcd6\") pod \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") "
Jul 31 15:25:51.728091 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.727924    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kserve-provision-location\") pod \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") "
Jul 31 15:25:51.728091 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.727943    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls\") pod \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") "
Jul 31 15:25:51.728091 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.727989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-rbac-proxy-sar-config\") pod \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") "
Jul 31 15:25:51.728091 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.728026    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-cabundle-cert\") pod \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\" (UID: \"8f2a4620-4da0-4fc8-8745-fc13f77402d7\") "
Jul 31 15:25:51.728349 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.728282    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "8f2a4620-4da0-4fc8-8745-fc13f77402d7" (UID: "8f2a4620-4da0-4fc8-8745-fc13f77402d7"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:25:51.728407 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.728388    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f2a4620-4da0-4fc8-8745-fc13f77402d7" (UID: "8f2a4620-4da0-4fc8-8745-fc13f77402d7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:25:51.728510 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.728480    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "8f2a4620-4da0-4fc8-8745-fc13f77402d7" (UID: "8f2a4620-4da0-4fc8-8745-fc13f77402d7"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:25:51.730065 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.730041    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f2a4620-4da0-4fc8-8745-fc13f77402d7" (UID: "8f2a4620-4da0-4fc8-8745-fc13f77402d7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:25:51.730272 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.730127    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-api-access-kgcd6" (OuterVolumeSpecName: "kube-api-access-kgcd6") pod "8f2a4620-4da0-4fc8-8745-fc13f77402d7" (UID: "8f2a4620-4da0-4fc8-8745-fc13f77402d7"). InnerVolumeSpecName "kube-api-access-kgcd6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:25:51.828958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.828870    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:25:51.828958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.828901    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f2a4620-4da0-4fc8-8745-fc13f77402d7-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:25:51.828958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.828911    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:25:51.828958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.828920    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/8f2a4620-4da0-4fc8-8745-fc13f77402d7-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:25:51.828958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.828930    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kgcd6\" (UniqueName: \"kubernetes.io/projected/8f2a4620-4da0-4fc8-8745-fc13f77402d7-kube-api-access-kgcd6\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:25:51.947934 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.947902    2561 generic.go:356] "Generic (PLEG): container finished" podID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerID="a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4" exitCode=0
Jul 31 15:25:51.948113 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.947964    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"
Jul 31 15:25:51.948113 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.947987    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" event={"ID":"8f2a4620-4da0-4fc8-8745-fc13f77402d7","Type":"ContainerDied","Data":"a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4"}
Jul 31 15:25:51.948113 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.948067    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd" event={"ID":"8f2a4620-4da0-4fc8-8745-fc13f77402d7","Type":"ContainerDied","Data":"023bab83cfbfe19ab485bb5780b8b81b90a97aa9684d58a53b1b9915264cee90"}
Jul 31 15:25:51.948113 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.948105    2561 scope.go:117] "RemoveContainer" containerID="81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5"
Jul 31 15:25:51.958106 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.958081    2561 scope.go:117] "RemoveContainer" containerID="a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4"
Jul 31 15:25:51.967945 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.967922    2561 scope.go:117] "RemoveContainer" containerID="309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b"
Jul 31 15:25:51.971547 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.971522    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"]
Jul 31 15:25:51.971887 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.971869    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kube-rbac-proxy"
Jul 31 15:25:51.971957 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.971893    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kube-rbac-proxy"
Jul 31 15:25:51.971957 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.971921    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="storage-initializer"
Jul 31 15:25:51.971957 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.971932    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="storage-initializer"
Jul 31 15:25:51.971957 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.971943    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container"
Jul 31 15:25:51.971957 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.971954    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container"
Jul 31 15:25:51.972250 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.972045    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kserve-container"
Jul 31 15:25:51.972250 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.972058    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" containerName="kube-rbac-proxy"
Jul 31 15:25:51.975837 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.975818    2561 scope.go:117] "RemoveContainer" containerID="81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5"
Jul 31 15:25:51.976129 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:25:51.976107    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5\": container with ID starting with 81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5 not found: ID does not exist" containerID="81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5"
Jul 31 15:25:51.976228 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.976140    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5"} err="failed to get container status \"81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5\": rpc error: code = NotFound desc = could not find container \"81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5\": container with ID starting with 81b8820784985a02fb73b38fe15f828e4bea5167cd5f0421f7dc928329fd40c5 not found: ID does not exist"
Jul 31 15:25:51.976228 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.976179    2561 scope.go:117] "RemoveContainer" containerID="a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4"
Jul 31 15:25:51.976451 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:25:51.976433    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4\": container with ID starting with a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4 not found: ID does not exist" containerID="a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4"
Jul 31 15:25:51.976508 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.976456    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4"} err="failed to get container status \"a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4\": rpc error: code = NotFound desc = could not find container \"a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4\": container with ID starting with a17126414b965d4f1535c08a450d4b10f806d4c975196cb24446eb6f16c9c9c4 not found: ID does not exist"
Jul 31 15:25:51.976671 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.976474    2561 scope.go:117] "RemoveContainer" containerID="309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b"
Jul 31 15:25:51.976908 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.976889    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:51.976908 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:25:51.976897    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b\": container with ID starting with 309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b not found: ID does not exist" containerID="309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b"
Jul 31 15:25:51.977020 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.976921    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b"} err="failed to get container status \"309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b\": rpc error: code = NotFound desc = could not find container \"309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b\": container with ID starting with 309209d6d35796420179bf1f975b63f6e67842fee82bd28a8dea7d6e00f6245b not found: ID does not exist"
Jul 31 15:25:51.979239 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.979217    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-predictor-serving-cert\""
Jul 31 15:25:51.979356 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.979294    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:25:51.979685 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.979668    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:25:51.980183 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.979886    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:25:51.980183 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.979928    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:25:51.980183 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.979999    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"]
Jul 31 15:25:51.980183 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.980043    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-canary-stop-kube-rbac-proxy-sar-config\""
Jul 31 15:25:51.980183 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.980125    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:25:51.988254 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.988229    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"]
Jul 31 15:25:51.991754 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:51.991726    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-rollback-predictor-7f985c86-6w4dd"]
Jul 31 15:25:52.031310 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.031280    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9467ce52-d688-40dd-a7de-3632a5c7c760-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.031516 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.031478    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.031598 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.031520    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-cabundle-cert\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.031598 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.031574    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.031699 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.031623    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8vrl\" (UniqueName: \"kubernetes.io/projected/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-api-access-t8vrl\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.133127 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.133038    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.133127 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.133072    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-cabundle-cert\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.133127 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.133095    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.133127 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.133121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8vrl\" (UniqueName: \"kubernetes.io/projected/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-api-access-t8vrl\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.133489 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.133227    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9467ce52-d688-40dd-a7de-3632a5c7c760-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.133489 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:25:52.133240    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 31 15:25:52.133489 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:25:52.133316    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls podName:9467ce52-d688-40dd-a7de-3632a5c7c760 nodeName:}" failed. No retries permitted until 2026-07-31 15:25:52.633294441 +0000 UTC m=+2249.152718471 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls") pod "isvc-canary-stop-predictor-6787f74994-mmx6c" (UID: "9467ce52-d688-40dd-a7de-3632a5c7c760") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 31 15:25:52.133654 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.133595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9467ce52-d688-40dd-a7de-3632a5c7c760-kserve-provision-location\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.133811 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.133792    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-rbac-proxy-sar-config\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.133954 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.133933    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-cabundle-cert\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.144594 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.144563    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8vrl\" (UniqueName: \"kubernetes.io/projected/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-api-access-t8vrl\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.637938 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:52.637906    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:52.638123 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:25:52.638044    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-canary-stop-predictor-serving-cert: secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 31 15:25:52.638123 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:25:52.638097    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls podName:9467ce52-d688-40dd-a7de-3632a5c7c760 nodeName:}" failed. No retries permitted until 2026-07-31 15:25:53.63808338 +0000 UTC m=+2250.157507397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls") pod "isvc-canary-stop-predictor-6787f74994-mmx6c" (UID: "9467ce52-d688-40dd-a7de-3632a5c7c760") : secret "isvc-canary-stop-predictor-serving-cert" not found
Jul 31 15:25:53.646470 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:53.646432    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:53.648925 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:53.648897    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls\") pod \"isvc-canary-stop-predictor-6787f74994-mmx6c\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") " pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:53.790918 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:53.790879    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:53.926869 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:53.926844    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"]
Jul 31 15:25:53.928955 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:25:53.928926    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9467ce52_d688_40dd_a7de_3632a5c7c760.slice/crio-afe0872226b350d223e14f71da7741dddf58daa78d44b71543f66f29d1feda45 WatchSource:0}: Error finding container afe0872226b350d223e14f71da7741dddf58daa78d44b71543f66f29d1feda45: Status 404 returned error can't find the container with id afe0872226b350d223e14f71da7741dddf58daa78d44b71543f66f29d1feda45
Jul 31 15:25:53.961499 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:53.961466    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f2a4620-4da0-4fc8-8745-fc13f77402d7" path="/var/lib/kubelet/pods/8f2a4620-4da0-4fc8-8745-fc13f77402d7/volumes"
Jul 31 15:25:53.963364 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:53.963334    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" event={"ID":"9467ce52-d688-40dd-a7de-3632a5c7c760","Type":"ContainerStarted","Data":"afe0872226b350d223e14f71da7741dddf58daa78d44b71543f66f29d1feda45"}
Jul 31 15:25:54.969684 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:54.969646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" event={"ID":"9467ce52-d688-40dd-a7de-3632a5c7c760","Type":"ContainerStarted","Data":"38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0"}
Jul 31 15:25:57.983703 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:57.983668    2561 generic.go:356] "Generic (PLEG): container finished" podID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerID="38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0" exitCode=0
Jul 31 15:25:57.984140 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:57.983723    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" event={"ID":"9467ce52-d688-40dd-a7de-3632a5c7c760","Type":"ContainerDied","Data":"38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0"}
Jul 31 15:25:58.990283 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:58.990251    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" event={"ID":"9467ce52-d688-40dd-a7de-3632a5c7c760","Type":"ContainerStarted","Data":"7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0"}
Jul 31 15:25:58.990664 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:58.990309    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" event={"ID":"9467ce52-d688-40dd-a7de-3632a5c7c760","Type":"ContainerStarted","Data":"a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71"}
Jul 31 15:25:58.990664 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:58.990595    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:58.990777 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:58.990749    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:25:58.992063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:58.992027    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:25:59.011021 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:59.010965    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podStartSLOduration=4.135236958 podStartE2EDuration="8.010946097s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.875709139s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:25:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:25:59.010277627 +0000 UTC m=+2255.529701669" watchObservedRunningTime="2026-07-31 15:25:59.010946097 +0000 UTC m=+2255.530370136"
Jul 31 15:25:59.995951 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:25:59.995910    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:26:05.000896 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:26:05.000860    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:26:05.001458 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:26:05.001431    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:26:15.001726 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:26:15.001686    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:26:25.002186 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:26:25.002129    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:26:35.002285 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:26:35.002240    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:26:45.002187 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:26:45.002132    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:26:55.002422 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:26:55.002378    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:27:05.001500 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:27:05.001459    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.37:8080: connect: connection refused"
Jul 31 15:27:15.002222 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:27:15.002191    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:28:07.046425 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:07.046389    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"]
Jul 31 15:28:07.047060 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:07.046650    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container" containerID="cri-o://a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71" gracePeriod=30
Jul 31 15:28:07.047060 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:07.046668    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kube-rbac-proxy" containerID="cri-o://7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0" gracePeriod=30
Jul 31 15:28:07.547953 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:07.547919    2561 generic.go:356] "Generic (PLEG): container finished" podID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerID="7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0" exitCode=2
Jul 31 15:28:07.548125 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:07.547992    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" event={"ID":"9467ce52-d688-40dd-a7de-3632a5c7c760","Type":"ContainerDied","Data":"7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0"}
Jul 31 15:28:09.996596 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:09.996547    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.37:8643/healthz\": dial tcp 10.133.0.37:8643: connect: connection refused"
Jul 31 15:28:12.097936 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.097911    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:28:12.249025 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.248989    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t8vrl\" (UniqueName: \"kubernetes.io/projected/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-api-access-t8vrl\") pod \"9467ce52-d688-40dd-a7de-3632a5c7c760\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") "
Jul 31 15:28:12.249237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.249040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-cabundle-cert\") pod \"9467ce52-d688-40dd-a7de-3632a5c7c760\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") "
Jul 31 15:28:12.249237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.249071    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-rbac-proxy-sar-config\") pod \"9467ce52-d688-40dd-a7de-3632a5c7c760\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") "
Jul 31 15:28:12.249237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.249096    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9467ce52-d688-40dd-a7de-3632a5c7c760-kserve-provision-location\") pod \"9467ce52-d688-40dd-a7de-3632a5c7c760\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") "
Jul 31 15:28:12.249237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.249120    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls\") pod \"9467ce52-d688-40dd-a7de-3632a5c7c760\" (UID: \"9467ce52-d688-40dd-a7de-3632a5c7c760\") "
Jul 31 15:28:12.249487 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.249466    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9467ce52-d688-40dd-a7de-3632a5c7c760-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9467ce52-d688-40dd-a7de-3632a5c7c760" (UID: "9467ce52-d688-40dd-a7de-3632a5c7c760"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:28:12.249560 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.249478    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9467ce52-d688-40dd-a7de-3632a5c7c760" (UID: "9467ce52-d688-40dd-a7de-3632a5c7c760"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:28:12.249560 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.249545    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "9467ce52-d688-40dd-a7de-3632a5c7c760" (UID: "9467ce52-d688-40dd-a7de-3632a5c7c760"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:28:12.251304 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.251281    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9467ce52-d688-40dd-a7de-3632a5c7c760" (UID: "9467ce52-d688-40dd-a7de-3632a5c7c760"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:28:12.251304 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.251288    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-api-access-t8vrl" (OuterVolumeSpecName: "kube-api-access-t8vrl") pod "9467ce52-d688-40dd-a7de-3632a5c7c760" (UID: "9467ce52-d688-40dd-a7de-3632a5c7c760"). InnerVolumeSpecName "kube-api-access-t8vrl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:28:12.350386 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.350347    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t8vrl\" (UniqueName: \"kubernetes.io/projected/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-api-access-t8vrl\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.350386 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.350379    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.350386 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.350390    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9467ce52-d688-40dd-a7de-3632a5c7c760-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.350386 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.350400    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9467ce52-d688-40dd-a7de-3632a5c7c760-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.350658 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.350409    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9467ce52-d688-40dd-a7de-3632a5c7c760-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:28:12.570776 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.570687    2561 generic.go:356] "Generic (PLEG): container finished" podID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerID="a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71" exitCode=0
Jul 31 15:28:12.570776 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.570764    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"
Jul 31 15:28:12.571002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.570771    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" event={"ID":"9467ce52-d688-40dd-a7de-3632a5c7c760","Type":"ContainerDied","Data":"a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71"}
Jul 31 15:28:12.571002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.570834    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c" event={"ID":"9467ce52-d688-40dd-a7de-3632a5c7c760","Type":"ContainerDied","Data":"afe0872226b350d223e14f71da7741dddf58daa78d44b71543f66f29d1feda45"}
Jul 31 15:28:12.571002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.570862    2561 scope.go:117] "RemoveContainer" containerID="7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0"
Jul 31 15:28:12.580386 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.580367    2561 scope.go:117] "RemoveContainer" containerID="a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71"
Jul 31 15:28:12.588783 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.588759    2561 scope.go:117] "RemoveContainer" containerID="38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0"
Jul 31 15:28:12.593036 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.593008    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"]
Jul 31 15:28:12.596583 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.596564    2561 scope.go:117] "RemoveContainer" containerID="7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0"
Jul 31 15:28:12.596883 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:28:12.596855    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0\": container with ID starting with 7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0 not found: ID does not exist" containerID="7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0"
Jul 31 15:28:12.596934 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.596903    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0"} err="failed to get container status \"7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0\": rpc error: code = NotFound desc = could not find container \"7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0\": container with ID starting with 7f073da0ef1e701d4e4ba7523549c98a25cdd00a749b34c24d8cfb8dba9190e0 not found: ID does not exist"
Jul 31 15:28:12.596934 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.596925    2561 scope.go:117] "RemoveContainer" containerID="a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71"
Jul 31 15:28:12.597190 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:28:12.597170    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71\": container with ID starting with a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71 not found: ID does not exist" containerID="a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71"
Jul 31 15:28:12.597295 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.597193    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71"} err="failed to get container status \"a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71\": rpc error: code = NotFound desc = could not find container \"a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71\": container with ID starting with a2d65a7ae2a5d95c9973a53f11ac359c7829c0222bae16d633add84631c8cd71 not found: ID does not exist"
Jul 31 15:28:12.597295 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.597208    2561 scope.go:117] "RemoveContainer" containerID="38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0"
Jul 31 15:28:12.597445 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:28:12.597430    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0\": container with ID starting with 38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0 not found: ID does not exist" containerID="38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0"
Jul 31 15:28:12.597490 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.597449    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0"} err="failed to get container status \"38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0\": rpc error: code = NotFound desc = could not find container \"38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0\": container with ID starting with 38784c47271c93a17e19ccbc99d0094e88f6d07c57a3f9edf6418cc695d04bf0 not found: ID does not exist"
Jul 31 15:28:12.602573 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:12.602548    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-canary-stop-predictor-6787f74994-mmx6c"]
Jul 31 15:28:13.961172 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:13.961124    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" path="/var/lib/kubelet/pods/9467ce52-d688-40dd-a7de-3632a5c7c760/volumes"
Jul 31 15:28:17.342721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.342683    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"]
Jul 31 15:28:17.343218 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.343025    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="storage-initializer"
Jul 31 15:28:17.343218 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.343037    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="storage-initializer"
Jul 31 15:28:17.343218 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.343045    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container"
Jul 31 15:28:17.343218 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.343052    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container"
Jul 31 15:28:17.343218 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.343077    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kube-rbac-proxy"
Jul 31 15:28:17.343218 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.343083    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kube-rbac-proxy"
Jul 31 15:28:17.343218 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.343132    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kube-rbac-proxy"
Jul 31 15:28:17.343218 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.343144    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9467ce52-d688-40dd-a7de-3632a5c7c760" containerName="kserve-container"
Jul 31 15:28:17.347885 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.347865    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.350300 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.350276    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-kube-rbac-proxy-sar-config\""
Jul 31 15:28:17.350436 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.350381    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:28:17.350624 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.350604    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:28:17.350750 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.350630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:28:17.351148 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.351127    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:28:17.351369 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.351132    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-predictor-serving-cert\""
Jul 31 15:28:17.351462 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.351226    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:28:17.356420 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.356390    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"]
Jul 31 15:28:17.495942 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.495909    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.496142 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.495965    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e9f476-e303-43be-bcf9-ed51f7447020-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.496142 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.495995    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.496142 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.496016    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk4hq\" (UniqueName: \"kubernetes.io/projected/99e9f476-e303-43be-bcf9-ed51f7447020-kube-api-access-xk4hq\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.496142 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.496052    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.597347 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.597208    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e9f476-e303-43be-bcf9-ed51f7447020-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.597347 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.597287    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.597347 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.597318    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk4hq\" (UniqueName: \"kubernetes.io/projected/99e9f476-e303-43be-bcf9-ed51f7447020-kube-api-access-xk4hq\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.597710 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.597476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.597710 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.597656    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.597813 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.597791    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e9f476-e303-43be-bcf9-ed51f7447020-kserve-provision-location\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.597869 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:28:17.597810    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-lightgbm-predictor-serving-cert: secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 31 15:28:17.597945 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:28:17.597931    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls podName:99e9f476-e303-43be-bcf9-ed51f7447020 nodeName:}" failed. No retries permitted until 2026-07-31 15:28:18.097903078 +0000 UTC m=+2394.617327109 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls") pod "isvc-lightgbm-predictor-54f85ddf88-n4c62" (UID: "99e9f476-e303-43be-bcf9-ed51f7447020") : secret "isvc-lightgbm-predictor-serving-cert" not found
Jul 31 15:28:17.598339 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.598318    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.598454 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.598395    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-cabundle-cert\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:17.610142 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:17.610089    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk4hq\" (UniqueName: \"kubernetes.io/projected/99e9f476-e303-43be-bcf9-ed51f7447020-kube-api-access-xk4hq\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:18.101997 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:18.101958    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:18.104464 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:18.104436    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls\") pod \"isvc-lightgbm-predictor-54f85ddf88-n4c62\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") " pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:18.262444 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:18.262403    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:18.394984 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:18.394909    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"]
Jul 31 15:28:18.397330 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:28:18.397302    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e9f476_e303_43be_bcf9_ed51f7447020.slice/crio-2169a34aed515f6701f1058f925df9fd0cafb70dea42ae9009c27f8ba0d21ed2 WatchSource:0}: Error finding container 2169a34aed515f6701f1058f925df9fd0cafb70dea42ae9009c27f8ba0d21ed2: Status 404 returned error can't find the container with id 2169a34aed515f6701f1058f925df9fd0cafb70dea42ae9009c27f8ba0d21ed2
Jul 31 15:28:18.601684 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:18.601646    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" event={"ID":"99e9f476-e303-43be-bcf9-ed51f7447020","Type":"ContainerStarted","Data":"980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85"}
Jul 31 15:28:18.601920 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:18.601708    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" event={"ID":"99e9f476-e303-43be-bcf9-ed51f7447020","Type":"ContainerStarted","Data":"2169a34aed515f6701f1058f925df9fd0cafb70dea42ae9009c27f8ba0d21ed2"}
Jul 31 15:28:22.622599 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:22.622510    2561 generic.go:356] "Generic (PLEG): container finished" podID="99e9f476-e303-43be-bcf9-ed51f7447020" containerID="980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85" exitCode=0
Jul 31 15:28:22.622599 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:22.622578    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" event={"ID":"99e9f476-e303-43be-bcf9-ed51f7447020","Type":"ContainerDied","Data":"980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85"}
Jul 31 15:28:22.623822 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:22.623807    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:28:24.089890 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:24.089858    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:28:24.094407 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:24.094378    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:28:29.659866 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:29.659832    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" event={"ID":"99e9f476-e303-43be-bcf9-ed51f7447020","Type":"ContainerStarted","Data":"81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa"}
Jul 31 15:28:29.660285 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:29.659891    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" event={"ID":"99e9f476-e303-43be-bcf9-ed51f7447020","Type":"ContainerStarted","Data":"ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da"}
Jul 31 15:28:29.660285 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:29.660078    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:29.683061 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:29.683008    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podStartSLOduration=2.614644916 podStartE2EDuration="12.682991407s" totalImagesPullingTime="6.21525368s" totalInitContainerRuntime="3.853092811s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:28:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:28:29.681491254 +0000 UTC m=+2406.200915306" watchObservedRunningTime="2026-07-31 15:28:29.682991407 +0000 UTC m=+2406.202415442"
Jul 31 15:28:30.664639 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:30.664607    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:30.665875 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:30.665847    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:28:31.668636 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:31.668597    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:28:36.674536 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:36.674509    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:28:36.675039 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:36.675010    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:28:46.675321 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:46.675229    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:28:56.675213 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:28:56.675149    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:29:06.675332 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:06.675295    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:29:16.675325 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:16.675285    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:29:26.675053 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:26.675011    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:29:36.675517 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:36.675461    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.38:8080: connect: connection refused"
Jul 31 15:29:46.676221 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:46.676190    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:29:48.036986 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:48.036954    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"]
Jul 31 15:29:48.037415 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:48.037197    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container" containerID="cri-o://ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da" gracePeriod=30
Jul 31 15:29:48.037415 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:48.037250    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kube-rbac-proxy" containerID="cri-o://81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa" gracePeriod=30
Jul 31 15:29:49.004879 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:49.004846    2561 generic.go:356] "Generic (PLEG): container finished" podID="99e9f476-e303-43be-bcf9-ed51f7447020" containerID="81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa" exitCode=2
Jul 31 15:29:49.005142 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:49.004927    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" event={"ID":"99e9f476-e303-43be-bcf9-ed51f7447020","Type":"ContainerDied","Data":"81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa"}
Jul 31 15:29:51.669591 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:51.669547    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.38:8643/healthz\": dial tcp 10.133.0.38:8643: connect: connection refused"
Jul 31 15:29:52.687237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.687213    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:29:52.748560 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.748469    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls\") pod \"99e9f476-e303-43be-bcf9-ed51f7447020\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") "
Jul 31 15:29:52.748560 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.748524    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xk4hq\" (UniqueName: \"kubernetes.io/projected/99e9f476-e303-43be-bcf9-ed51f7447020-kube-api-access-xk4hq\") pod \"99e9f476-e303-43be-bcf9-ed51f7447020\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") "
Jul 31 15:29:52.748560 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.748556    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e9f476-e303-43be-bcf9-ed51f7447020-kserve-provision-location\") pod \"99e9f476-e303-43be-bcf9-ed51f7447020\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") "
Jul 31 15:29:52.748856 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.748588    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-cabundle-cert\") pod \"99e9f476-e303-43be-bcf9-ed51f7447020\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") "
Jul 31 15:29:52.748856 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.748789    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-kube-rbac-proxy-sar-config\") pod \"99e9f476-e303-43be-bcf9-ed51f7447020\" (UID: \"99e9f476-e303-43be-bcf9-ed51f7447020\") "
Jul 31 15:29:52.749007 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.748982    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/99e9f476-e303-43be-bcf9-ed51f7447020-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "99e9f476-e303-43be-bcf9-ed51f7447020" (UID: "99e9f476-e303-43be-bcf9-ed51f7447020"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:29:52.749135 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.749052    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "99e9f476-e303-43be-bcf9-ed51f7447020" (UID: "99e9f476-e303-43be-bcf9-ed51f7447020"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:29:52.749135 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.749084    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "99e9f476-e303-43be-bcf9-ed51f7447020" (UID: "99e9f476-e303-43be-bcf9-ed51f7447020"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:29:52.749135 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.749112    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/99e9f476-e303-43be-bcf9-ed51f7447020-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:29:52.750713 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.750690    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "99e9f476-e303-43be-bcf9-ed51f7447020" (UID: "99e9f476-e303-43be-bcf9-ed51f7447020"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:29:52.750771 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.750698    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e9f476-e303-43be-bcf9-ed51f7447020-kube-api-access-xk4hq" (OuterVolumeSpecName: "kube-api-access-xk4hq") pod "99e9f476-e303-43be-bcf9-ed51f7447020" (UID: "99e9f476-e303-43be-bcf9-ed51f7447020"). InnerVolumeSpecName "kube-api-access-xk4hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:29:52.849656 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.849616    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:29:52.849656 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.849647    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/99e9f476-e303-43be-bcf9-ed51f7447020-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:29:52.849656 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.849657    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xk4hq\" (UniqueName: \"kubernetes.io/projected/99e9f476-e303-43be-bcf9-ed51f7447020-kube-api-access-xk4hq\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:29:52.849656 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:52.849665    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/99e9f476-e303-43be-bcf9-ed51f7447020-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:29:53.023751 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.023667    2561 generic.go:356] "Generic (PLEG): container finished" podID="99e9f476-e303-43be-bcf9-ed51f7447020" containerID="ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da" exitCode=0
Jul 31 15:29:53.023751 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.023743    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"
Jul 31 15:29:53.023980 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.023763    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" event={"ID":"99e9f476-e303-43be-bcf9-ed51f7447020","Type":"ContainerDied","Data":"ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da"}
Jul 31 15:29:53.023980 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.023838    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62" event={"ID":"99e9f476-e303-43be-bcf9-ed51f7447020","Type":"ContainerDied","Data":"2169a34aed515f6701f1058f925df9fd0cafb70dea42ae9009c27f8ba0d21ed2"}
Jul 31 15:29:53.023980 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.023878    2561 scope.go:117] "RemoveContainer" containerID="81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa"
Jul 31 15:29:53.032564 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.032546    2561 scope.go:117] "RemoveContainer" containerID="ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da"
Jul 31 15:29:53.040070 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.040046    2561 scope.go:117] "RemoveContainer" containerID="980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85"
Jul 31 15:29:53.046068 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.046047    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"]
Jul 31 15:29:53.047574 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.047557    2561 scope.go:117] "RemoveContainer" containerID="81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa"
Jul 31 15:29:53.047838 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:29:53.047820    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa\": container with ID starting with 81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa not found: ID does not exist" containerID="81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa"
Jul 31 15:29:53.047891 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.047845    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa"} err="failed to get container status \"81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa\": rpc error: code = NotFound desc = could not find container \"81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa\": container with ID starting with 81b5311daa6174fd4b538fb3cde7d751cbd2b8006dc947998851a4834e46b1aa not found: ID does not exist"
Jul 31 15:29:53.047891 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.047862    2561 scope.go:117] "RemoveContainer" containerID="ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da"
Jul 31 15:29:53.048099 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:29:53.048080    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da\": container with ID starting with ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da not found: ID does not exist" containerID="ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da"
Jul 31 15:29:53.048135 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.048108    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da"} err="failed to get container status \"ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da\": rpc error: code = NotFound desc = could not find container \"ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da\": container with ID starting with ff87ad4a13494b42a65f94125f144a865d6ee90641a60ca3b0000890f03a38da not found: ID does not exist"
Jul 31 15:29:53.048135 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.048126    2561 scope.go:117] "RemoveContainer" containerID="980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85"
Jul 31 15:29:53.048387 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:29:53.048367    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85\": container with ID starting with 980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85 not found: ID does not exist" containerID="980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85"
Jul 31 15:29:53.048432 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.048392    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85"} err="failed to get container status \"980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85\": rpc error: code = NotFound desc = could not find container \"980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85\": container with ID starting with 980e90d085675fabec09bcb2c1dbcc275002c4229f18241e136deef5c2bc9d85 not found: ID does not exist"
Jul 31 15:29:53.053538 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.053519    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-predictor-54f85ddf88-n4c62"]
Jul 31 15:29:53.195556 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.195516    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"]
Jul 31 15:29:53.195912 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.195895    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="storage-initializer"
Jul 31 15:29:53.195999 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.195915    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="storage-initializer"
Jul 31 15:29:53.195999 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.195928    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kube-rbac-proxy"
Jul 31 15:29:53.195999 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.195936    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kube-rbac-proxy"
Jul 31 15:29:53.195999 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.195970    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container"
Jul 31 15:29:53.195999 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.195980    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container"
Jul 31 15:29:53.196307 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.196056    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kserve-container"
Jul 31 15:29:53.196307 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.196070    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" containerName="kube-rbac-proxy"
Jul 31 15:29:53.200561 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.200540    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.205899 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.205875    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-predictor-serving-cert\""
Jul 31 15:29:53.205899 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.205889    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:29:53.206102 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.205924    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:29:53.206102 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.205897    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:29:53.206102 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.205975    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:29:53.206296 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.206282    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:29:53.206360 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.206310    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:29:53.213339 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.213317    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"]
Jul 31 15:29:53.252746 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.252716    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.252929 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.252766    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cdswl\" (UniqueName: \"kubernetes.io/projected/2ed95582-b807-45af-a61b-a299d8057713-kube-api-access-cdswl\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.252929 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.252825    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.253074 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.252926    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed95582-b807-45af-a61b-a299d8057713-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.253074 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.252961    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed95582-b807-45af-a61b-a299d8057713-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.353771 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.353684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed95582-b807-45af-a61b-a299d8057713-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.353771 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.353719    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed95582-b807-45af-a61b-a299d8057713-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.353993 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.353841    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.353993 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.353892    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cdswl\" (UniqueName: \"kubernetes.io/projected/2ed95582-b807-45af-a61b-a299d8057713-kube-api-access-cdswl\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.353993 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.353955    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.354211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.354063    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed95582-b807-45af-a61b-a299d8057713-kserve-provision-location\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.354538 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.354517    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-cabundle-cert\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.354617 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.354581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.356197 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.356180    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed95582-b807-45af-a61b-a299d8057713-proxy-tls\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.366620 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.366599    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cdswl\" (UniqueName: \"kubernetes.io/projected/2ed95582-b807-45af-a61b-a299d8057713-kube-api-access-cdswl\") pod \"isvc-lightgbm-runtime-predictor-577f7db7cb-522kn\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") " pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.513907 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.513837    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:53.667508 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.667483    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"]
Jul 31 15:29:53.669793 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:29:53.669763    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ed95582_b807_45af_a61b_a299d8057713.slice/crio-8a37f858cf817084d16f7d34a02bf1c6a7626a650ccd29eb795db74b34b6e24e WatchSource:0}: Error finding container 8a37f858cf817084d16f7d34a02bf1c6a7626a650ccd29eb795db74b34b6e24e: Status 404 returned error can't find the container with id 8a37f858cf817084d16f7d34a02bf1c6a7626a650ccd29eb795db74b34b6e24e
Jul 31 15:29:53.961184 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:53.961099    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99e9f476-e303-43be-bcf9-ed51f7447020" path="/var/lib/kubelet/pods/99e9f476-e303-43be-bcf9-ed51f7447020/volumes"
Jul 31 15:29:54.030134 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:54.030092    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" event={"ID":"2ed95582-b807-45af-a61b-a299d8057713","Type":"ContainerStarted","Data":"5eb26b9bc891a04972247e4973e708aac496ea03bf2218f3bd65a3cb82a0ff07"}
Jul 31 15:29:54.030350 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:54.030190    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" event={"ID":"2ed95582-b807-45af-a61b-a299d8057713","Type":"ContainerStarted","Data":"8a37f858cf817084d16f7d34a02bf1c6a7626a650ccd29eb795db74b34b6e24e"}
Jul 31 15:29:58.050231 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:58.050195    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ed95582-b807-45af-a61b-a299d8057713" containerID="5eb26b9bc891a04972247e4973e708aac496ea03bf2218f3bd65a3cb82a0ff07" exitCode=0
Jul 31 15:29:58.050604 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:58.050273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" event={"ID":"2ed95582-b807-45af-a61b-a299d8057713","Type":"ContainerDied","Data":"5eb26b9bc891a04972247e4973e708aac496ea03bf2218f3bd65a3cb82a0ff07"}
Jul 31 15:29:59.058071 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:59.058032    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" event={"ID":"2ed95582-b807-45af-a61b-a299d8057713","Type":"ContainerStarted","Data":"cfafa81cd8d643d6399be40e6fd81b1bbc51c0bb7052a2f9b076aff48106ae97"}
Jul 31 15:29:59.058570 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:59.058109    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" event={"ID":"2ed95582-b807-45af-a61b-a299d8057713","Type":"ContainerStarted","Data":"d0a01db3019c5b7baab846808c41673c1ade732c979bcdef4d0e547818f851e0"}
Jul 31 15:29:59.058570 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:59.058327    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:29:59.081177 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:29:59.081114    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podStartSLOduration=2.527215221 podStartE2EDuration="6.081095583s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.553880362s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:29:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:29:59.080221246 +0000 UTC m=+2495.599645288" watchObservedRunningTime="2026-07-31 15:29:59.081095583 +0000 UTC m=+2495.600519621"
Jul 31 15:30:00.062663 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:00.062636    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:30:00.064211 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:00.064183    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:30:01.067269 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:01.067228    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:30:06.072149 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:06.072118    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:30:06.072693 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:06.072666    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:30:16.072668 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:16.072580    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:30:26.072705 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:26.072662    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:30:36.072717 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:36.072678    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:30:46.072747 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:46.072698    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:30:56.072974 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:30:56.072933    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:31:06.072666 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:06.072621    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:31:14.957055 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:14.957015    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:31:23.590345 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:23.590310    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"]
Jul 31 15:31:23.590767 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:23.590570    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" containerID="cri-o://d0a01db3019c5b7baab846808c41673c1ade732c979bcdef4d0e547818f851e0" gracePeriod=30
Jul 31 15:31:23.590767 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:23.590611    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kube-rbac-proxy" containerID="cri-o://cfafa81cd8d643d6399be40e6fd81b1bbc51c0bb7052a2f9b076aff48106ae97" gracePeriod=30
Jul 31 15:31:24.437788 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:24.437751    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ed95582-b807-45af-a61b-a299d8057713" containerID="cfafa81cd8d643d6399be40e6fd81b1bbc51c0bb7052a2f9b076aff48106ae97" exitCode=2
Jul 31 15:31:24.437979 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:24.437807    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" event={"ID":"2ed95582-b807-45af-a61b-a299d8057713","Type":"ContainerDied","Data":"cfafa81cd8d643d6399be40e6fd81b1bbc51c0bb7052a2f9b076aff48106ae97"}
Jul 31 15:31:24.956675 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:24.956636    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.39:8080: connect: connection refused"
Jul 31 15:31:26.068382 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:26.068340    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.39:8643/healthz\": dial tcp 10.133.0.39:8643: connect: connection refused"
Jul 31 15:31:28.457223 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.457191    2561 generic.go:356] "Generic (PLEG): container finished" podID="2ed95582-b807-45af-a61b-a299d8057713" containerID="d0a01db3019c5b7baab846808c41673c1ade732c979bcdef4d0e547818f851e0" exitCode=0
Jul 31 15:31:28.457664 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.457269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" event={"ID":"2ed95582-b807-45af-a61b-a299d8057713","Type":"ContainerDied","Data":"d0a01db3019c5b7baab846808c41673c1ade732c979bcdef4d0e547818f851e0"}
Jul 31 15:31:28.548393 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.548367    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:31:28.689879 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.689782    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed95582-b807-45af-a61b-a299d8057713-kserve-provision-location\") pod \"2ed95582-b807-45af-a61b-a299d8057713\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") "
Jul 31 15:31:28.690050 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.689892    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-cabundle-cert\") pod \"2ed95582-b807-45af-a61b-a299d8057713\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") "
Jul 31 15:31:28.690050 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.689924    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed95582-b807-45af-a61b-a299d8057713-proxy-tls\") pod \"2ed95582-b807-45af-a61b-a299d8057713\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") "
Jul 31 15:31:28.690050 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.689941    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cdswl\" (UniqueName: \"kubernetes.io/projected/2ed95582-b807-45af-a61b-a299d8057713-kube-api-access-cdswl\") pod \"2ed95582-b807-45af-a61b-a299d8057713\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") "
Jul 31 15:31:28.690050 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.689957    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-kube-rbac-proxy-sar-config\") pod \"2ed95582-b807-45af-a61b-a299d8057713\" (UID: \"2ed95582-b807-45af-a61b-a299d8057713\") "
Jul 31 15:31:28.690327 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.690277    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2ed95582-b807-45af-a61b-a299d8057713-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "2ed95582-b807-45af-a61b-a299d8057713" (UID: "2ed95582-b807-45af-a61b-a299d8057713"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:31:28.690415 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.690361    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "2ed95582-b807-45af-a61b-a299d8057713" (UID: "2ed95582-b807-45af-a61b-a299d8057713"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:31:28.690492 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.690462    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "2ed95582-b807-45af-a61b-a299d8057713" (UID: "2ed95582-b807-45af-a61b-a299d8057713"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:31:28.692121 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.692102    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ed95582-b807-45af-a61b-a299d8057713-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "2ed95582-b807-45af-a61b-a299d8057713" (UID: "2ed95582-b807-45af-a61b-a299d8057713"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:31:28.692217 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.692114    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ed95582-b807-45af-a61b-a299d8057713-kube-api-access-cdswl" (OuterVolumeSpecName: "kube-api-access-cdswl") pod "2ed95582-b807-45af-a61b-a299d8057713" (UID: "2ed95582-b807-45af-a61b-a299d8057713"). InnerVolumeSpecName "kube-api-access-cdswl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:31:28.791339 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.791299    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:31:28.791339 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.791332    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/2ed95582-b807-45af-a61b-a299d8057713-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:31:28.791339 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.791341    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cdswl\" (UniqueName: \"kubernetes.io/projected/2ed95582-b807-45af-a61b-a299d8057713-kube-api-access-cdswl\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:31:28.791339 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.791351    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/2ed95582-b807-45af-a61b-a299d8057713-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:31:28.791716 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:28.791360    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/2ed95582-b807-45af-a61b-a299d8057713-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:31:29.463963 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:29.463925    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn" event={"ID":"2ed95582-b807-45af-a61b-a299d8057713","Type":"ContainerDied","Data":"8a37f858cf817084d16f7d34a02bf1c6a7626a650ccd29eb795db74b34b6e24e"}
Jul 31 15:31:29.464439 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:29.463975    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"
Jul 31 15:31:29.464439 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:29.463997    2561 scope.go:117] "RemoveContainer" containerID="cfafa81cd8d643d6399be40e6fd81b1bbc51c0bb7052a2f9b076aff48106ae97"
Jul 31 15:31:29.473435 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:29.473412    2561 scope.go:117] "RemoveContainer" containerID="d0a01db3019c5b7baab846808c41673c1ade732c979bcdef4d0e547818f851e0"
Jul 31 15:31:29.481125 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:29.481103    2561 scope.go:117] "RemoveContainer" containerID="5eb26b9bc891a04972247e4973e708aac496ea03bf2218f3bd65a3cb82a0ff07"
Jul 31 15:31:29.490788 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:29.490766    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"]
Jul 31 15:31:29.494358 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:29.494331    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-runtime-predictor-577f7db7cb-522kn"]
Jul 31 15:31:29.960700 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:31:29.960665    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ed95582-b807-45af-a61b-a299d8057713" path="/var/lib/kubelet/pods/2ed95582-b807-45af-a61b-a299d8057713/volumes"
Jul 31 15:32:24.040613 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.040577    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"]
Jul 31 15:32:24.041398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.041071    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container"
Jul 31 15:32:24.041398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.041089    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container"
Jul 31 15:32:24.041398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.041101    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="storage-initializer"
Jul 31 15:32:24.041398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.041110    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="storage-initializer"
Jul 31 15:32:24.041398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.041130    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kube-rbac-proxy"
Jul 31 15:32:24.041398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.041138    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kube-rbac-proxy"
Jul 31 15:32:24.041398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.041246    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kube-rbac-proxy"
Jul 31 15:32:24.041398 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.041261    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ed95582-b807-45af-a61b-a299d8057713" containerName="kserve-container"
Jul 31 15:32:24.044715 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.044693    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.048082 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.048059    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:32:24.048224 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.048185    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:32:24.048409 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.048394    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:32:24.048493 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.048440    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 31 15:32:24.048898 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.048873    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-lightgbm-v2-kserve-predictor-serving-cert\""
Jul 31 15:32:24.049041 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.049021    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:32:24.049109 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.049040    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:32:24.055125 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.055106    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"]
Jul 31 15:32:24.174564 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.174525    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-slqwh\" (UniqueName: \"kubernetes.io/projected/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-api-access-slqwh\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.174764 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.174573    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.180178 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.174604    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ebd071-2213-41cc-b66d-4463d021d7cf-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.180178 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.175344    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ebd071-2213-41cc-b66d-4463d021d7cf-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.180178 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.175499    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.276460 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.276429    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-slqwh\" (UniqueName: \"kubernetes.io/projected/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-api-access-slqwh\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.276646 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.276476    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.276646 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.276507    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ebd071-2213-41cc-b66d-4463d021d7cf-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.276646 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.276544    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ebd071-2213-41cc-b66d-4463d021d7cf-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.276646 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.276589    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.276965 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.276937    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ebd071-2213-41cc-b66d-4463d021d7cf-kserve-provision-location\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.277305 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.277284    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-cabundle-cert\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.277359 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.277309    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-rbac-proxy-sar-config\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.278844 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.278821    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ebd071-2213-41cc-b66d-4463d021d7cf-proxy-tls\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.286178 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.286144    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-slqwh\" (UniqueName: \"kubernetes.io/projected/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-api-access-slqwh\") pod \"isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") " pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.356574 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.356483    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:24.492015 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.491991    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"]
Jul 31 15:32:24.494549 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:32:24.494517    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ebd071_2213_41cc_b66d_4463d021d7cf.slice/crio-b772f0e1f9b9ea92c71dc052eec711652967cb0c091e1a16098a4bb0e37668a0 WatchSource:0}: Error finding container b772f0e1f9b9ea92c71dc052eec711652967cb0c091e1a16098a4bb0e37668a0: Status 404 returned error can't find the container with id b772f0e1f9b9ea92c71dc052eec711652967cb0c091e1a16098a4bb0e37668a0
Jul 31 15:32:24.695523 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.695419    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" event={"ID":"d4ebd071-2213-41cc-b66d-4463d021d7cf","Type":"ContainerStarted","Data":"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366"}
Jul 31 15:32:24.695523 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:24.695485    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" event={"ID":"d4ebd071-2213-41cc-b66d-4463d021d7cf","Type":"ContainerStarted","Data":"b772f0e1f9b9ea92c71dc052eec711652967cb0c091e1a16098a4bb0e37668a0"}
Jul 31 15:32:28.713715 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:28.713617    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerID="833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366" exitCode=0
Jul 31 15:32:28.713715 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:28.713688    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" event={"ID":"d4ebd071-2213-41cc-b66d-4463d021d7cf","Type":"ContainerDied","Data":"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366"}
Jul 31 15:32:29.722048 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:29.722002    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" event={"ID":"d4ebd071-2213-41cc-b66d-4463d021d7cf","Type":"ContainerStarted","Data":"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e"}
Jul 31 15:32:29.722478 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:29.722091    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" event={"ID":"d4ebd071-2213-41cc-b66d-4463d021d7cf","Type":"ContainerStarted","Data":"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e"}
Jul 31 15:32:29.722478 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:29.722407    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:29.722478 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:29.722463    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:29.724035 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:29.724008    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:32:29.747289 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:29.747240    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" podStartSLOduration=1.889038914 podStartE2EDuration="5.747222188s" totalImagesPullingTime="0s" totalInitContainerRuntime="3.858183274s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:32:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:32:29.745647831 +0000 UTC m=+2646.265071879" watchObservedRunningTime="2026-07-31 15:32:29.747222188 +0000 UTC m=+2646.266646226"
Jul 31 15:32:30.726064 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:30.726016    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:32:35.730756 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:35.730728    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:35.731279 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:35.731248    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.40:8080: connect: connection refused"
Jul 31 15:32:45.731958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:45.731924    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:54.154141 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.154100    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"]
Jul 31 15:32:54.154651 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.154405    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kserve-container" containerID="cri-o://a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e" gracePeriod=30
Jul 31 15:32:54.154651 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.154446    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kube-rbac-proxy" containerID="cri-o://d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e" gracePeriod=30
Jul 31 15:32:54.799420 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.799393    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:54.828480 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.828440    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerID="d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e" exitCode=2
Jul 31 15:32:54.828480 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.828475    2561 generic.go:356] "Generic (PLEG): container finished" podID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerID="a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e" exitCode=0
Jul 31 15:32:54.828754 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.828509    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"
Jul 31 15:32:54.828754 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.828538    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" event={"ID":"d4ebd071-2213-41cc-b66d-4463d021d7cf","Type":"ContainerDied","Data":"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e"}
Jul 31 15:32:54.828754 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.828616    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" event={"ID":"d4ebd071-2213-41cc-b66d-4463d021d7cf","Type":"ContainerDied","Data":"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e"}
Jul 31 15:32:54.828754 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.828653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt" event={"ID":"d4ebd071-2213-41cc-b66d-4463d021d7cf","Type":"ContainerDied","Data":"b772f0e1f9b9ea92c71dc052eec711652967cb0c091e1a16098a4bb0e37668a0"}
Jul 31 15:32:54.828754 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.828694    2561 scope.go:117] "RemoveContainer" containerID="d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e"
Jul 31 15:32:54.838689 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.838668    2561 scope.go:117] "RemoveContainer" containerID="a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e"
Jul 31 15:32:54.846284 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.846267    2561 scope.go:117] "RemoveContainer" containerID="833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366"
Jul 31 15:32:54.853977 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.853959    2561 scope.go:117] "RemoveContainer" containerID="d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e"
Jul 31 15:32:54.854299 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:32:54.854281    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e\": container with ID starting with d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e not found: ID does not exist" containerID="d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e"
Jul 31 15:32:54.854371 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.854308    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e"} err="failed to get container status \"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e\": rpc error: code = NotFound desc = could not find container \"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e\": container with ID starting with d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e not found: ID does not exist"
Jul 31 15:32:54.854371 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.854327    2561 scope.go:117] "RemoveContainer" containerID="a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e"
Jul 31 15:32:54.854527 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:32:54.854512    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e\": container with ID starting with a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e not found: ID does not exist" containerID="a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e"
Jul 31 15:32:54.854571 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.854530    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e"} err="failed to get container status \"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e\": rpc error: code = NotFound desc = could not find container \"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e\": container with ID starting with a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e not found: ID does not exist"
Jul 31 15:32:54.854571 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.854541    2561 scope.go:117] "RemoveContainer" containerID="833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366"
Jul 31 15:32:54.854739 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:32:54.854721    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366\": container with ID starting with 833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366 not found: ID does not exist" containerID="833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366"
Jul 31 15:32:54.854780 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.854743    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366"} err="failed to get container status \"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366\": rpc error: code = NotFound desc = could not find container \"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366\": container with ID starting with 833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366 not found: ID does not exist"
Jul 31 15:32:54.854780 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.854756    2561 scope.go:117] "RemoveContainer" containerID="d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e"
Jul 31 15:32:54.854947 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.854931    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e"} err="failed to get container status \"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e\": rpc error: code = NotFound desc = could not find container \"d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e\": container with ID starting with d134bc84d4eeec89f60e8e36670611e888e446b1026031076ec8c37bd6aecf5e not found: ID does not exist"
Jul 31 15:32:54.854992 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.854950    2561 scope.go:117] "RemoveContainer" containerID="a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e"
Jul 31 15:32:54.855121 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.855102    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e"} err="failed to get container status \"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e\": rpc error: code = NotFound desc = could not find container \"a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e\": container with ID starting with a5b44827860d396f8e20efafe89fe9ede7830581179c83246070dd886c13e19e not found: ID does not exist"
Jul 31 15:32:54.855193 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.855121    2561 scope.go:117] "RemoveContainer" containerID="833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366"
Jul 31 15:32:54.855363 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.855339    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366"} err="failed to get container status \"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366\": rpc error: code = NotFound desc = could not find container \"833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366\": container with ID starting with 833f1f1d5607e7217f130f101747ed72a0b89204bbda8ef47c95fdadebe18366 not found: ID does not exist"
Jul 31 15:32:54.939925 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.939832    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-rbac-proxy-sar-config\") pod \"d4ebd071-2213-41cc-b66d-4463d021d7cf\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") "
Jul 31 15:32:54.939925 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.939896    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ebd071-2213-41cc-b66d-4463d021d7cf-proxy-tls\") pod \"d4ebd071-2213-41cc-b66d-4463d021d7cf\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") "
Jul 31 15:32:54.940201 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.939933    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-cabundle-cert\") pod \"d4ebd071-2213-41cc-b66d-4463d021d7cf\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") "
Jul 31 15:32:54.940201 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.939975    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ebd071-2213-41cc-b66d-4463d021d7cf-kserve-provision-location\") pod \"d4ebd071-2213-41cc-b66d-4463d021d7cf\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") "
Jul 31 15:32:54.940201 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.940010    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-slqwh\" (UniqueName: \"kubernetes.io/projected/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-api-access-slqwh\") pod \"d4ebd071-2213-41cc-b66d-4463d021d7cf\" (UID: \"d4ebd071-2213-41cc-b66d-4463d021d7cf\") "
Jul 31 15:32:54.940365 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.940252    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d4ebd071-2213-41cc-b66d-4463d021d7cf" (UID: "d4ebd071-2213-41cc-b66d-4463d021d7cf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:32:54.940365 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.940353    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "d4ebd071-2213-41cc-b66d-4463d021d7cf" (UID: "d4ebd071-2213-41cc-b66d-4463d021d7cf"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:32:54.940445 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.940384    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4ebd071-2213-41cc-b66d-4463d021d7cf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "d4ebd071-2213-41cc-b66d-4463d021d7cf" (UID: "d4ebd071-2213-41cc-b66d-4463d021d7cf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:32:54.942063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.942044    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ebd071-2213-41cc-b66d-4463d021d7cf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d4ebd071-2213-41cc-b66d-4463d021d7cf" (UID: "d4ebd071-2213-41cc-b66d-4463d021d7cf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:32:54.942186 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:54.942139    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-api-access-slqwh" (OuterVolumeSpecName: "kube-api-access-slqwh") pod "d4ebd071-2213-41cc-b66d-4463d021d7cf" (UID: "d4ebd071-2213-41cc-b66d-4463d021d7cf"). InnerVolumeSpecName "kube-api-access-slqwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:32:55.041276 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:55.041232    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d4ebd071-2213-41cc-b66d-4463d021d7cf-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:32:55.041276 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:55.041272    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:32:55.041276 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:55.041286    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/d4ebd071-2213-41cc-b66d-4463d021d7cf-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:32:55.041521 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:55.041300    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-slqwh\" (UniqueName: \"kubernetes.io/projected/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-api-access-slqwh\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:32:55.041521 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:55.041314    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d4ebd071-2213-41cc-b66d-4463d021d7cf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:32:55.163432 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:55.163395    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"]
Jul 31 15:32:55.169377 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:55.169349    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-lightgbm-v2-kserve-predictor-5c8bcb4bd5-tzbkt"]
Jul 31 15:32:55.960615 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:55.960580    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" path="/var/lib/kubelet/pods/d4ebd071-2213-41cc-b66d-4463d021d7cf/volumes"
Jul 31 15:32:56.831582 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.831541    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"]
Jul 31 15:32:56.831947 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.831918    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kserve-container"
Jul 31 15:32:56.831947 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.831930    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kserve-container"
Jul 31 15:32:56.831947 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.831944    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="storage-initializer"
Jul 31 15:32:56.832049 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.831949    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="storage-initializer"
Jul 31 15:32:56.832049 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.831958    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kube-rbac-proxy"
Jul 31 15:32:56.832049 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.831964    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kube-rbac-proxy"
Jul 31 15:32:56.832049 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.832022    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kserve-container"
Jul 31 15:32:56.832049 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.832032    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ebd071-2213-41cc-b66d-4463d021d7cf" containerName="kube-rbac-proxy"
Jul 31 15:32:56.836740 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.836721    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:56.839465 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.839440    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-predictor-serving-cert\""
Jul 31 15:32:56.840064 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.840047    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:32:56.840599 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.840573    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:32:56.840721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.840633    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:32:56.840721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.840646    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-xgboost-v2-kube-rbac-proxy-sar-config\""
Jul 31 15:32:56.840721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.840652    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:32:56.841046 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.841028    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:32:56.848059 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.848041    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"]
Jul 31 15:32:56.959584 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.959546    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:56.959796 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.959606    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:56.959796 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.959711    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fptc7\" (UniqueName: \"kubernetes.io/projected/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-api-access-fptc7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:56.959796 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.959774    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:56.959949 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:56.959824    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.061043 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.060991    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fptc7\" (UniqueName: \"kubernetes.io/projected/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-api-access-fptc7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.061256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.061071    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.061256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.061121    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.061256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.061179    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.061256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.061235    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.061731 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.061702    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kserve-provision-location\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.061899 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.061881    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-cabundle-cert\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.061964 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.061902    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-rbac-proxy-sar-config\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.063518 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.063497    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-proxy-tls\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.070616 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.070595    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fptc7\" (UniqueName: \"kubernetes.io/projected/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-api-access-fptc7\") pod \"isvc-xgboost-v2-predictor-fcc668bc-wwqb7\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") " pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.147740 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.147645    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:32:57.284654 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.284626    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"]
Jul 31 15:32:57.286414 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:32:57.286379    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddca8f3c1_aa49_4241_9a7d_e9672fb41ac5.slice/crio-511b2c18882fd3c939e7cbc8ffda3a93222fcaee2a551bc6c86125070bd2cfff WatchSource:0}: Error finding container 511b2c18882fd3c939e7cbc8ffda3a93222fcaee2a551bc6c86125070bd2cfff: Status 404 returned error can't find the container with id 511b2c18882fd3c939e7cbc8ffda3a93222fcaee2a551bc6c86125070bd2cfff
Jul 31 15:32:57.846235 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.846196    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" event={"ID":"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5","Type":"ContainerStarted","Data":"c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326"}
Jul 31 15:32:57.846612 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:32:57.846269    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" event={"ID":"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5","Type":"ContainerStarted","Data":"511b2c18882fd3c939e7cbc8ffda3a93222fcaee2a551bc6c86125070bd2cfff"}
Jul 31 15:33:01.866435 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:01.866395    2561 generic.go:356] "Generic (PLEG): container finished" podID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerID="c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326" exitCode=0
Jul 31 15:33:01.866435 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:01.866435    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" event={"ID":"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5","Type":"ContainerDied","Data":"c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326"}
Jul 31 15:33:20.967324 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:20.967231    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" event={"ID":"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5","Type":"ContainerStarted","Data":"6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4"}
Jul 31 15:33:20.967324 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:20.967296    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" event={"ID":"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5","Type":"ContainerStarted","Data":"f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8"}
Jul 31 15:33:20.967724 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:20.967486    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:33:20.988272 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:20.988219    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podStartSLOduration=2.534829657 podStartE2EDuration="24.988200723s" totalImagesPullingTime="18.659960777s" totalInitContainerRuntime="3.793410289s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:32:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:33:20.986841704 +0000 UTC m=+2697.506265743" watchObservedRunningTime="2026-07-31 15:33:20.988200723 +0000 UTC m=+2697.507624760"
Jul 31 15:33:21.971876 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:21.971846    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:33:21.973072 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:21.973042    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:33:22.976477 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:22.976440    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:33:24.121547 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:24.121517    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:33:24.126310 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:24.126281    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:33:27.982322 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:27.982288    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:33:27.982948 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:27.982922    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:33:37.983413 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:37.983365    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:33:47.983493 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:47.983444    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:33:57.983767 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:33:57.983721    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:34:07.983773 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:07.983713    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:34:17.983254 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:17.983136    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:34:27.984253 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:27.984218    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:34:36.873369 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:36.873333    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"]
Jul 31 15:34:36.873786 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:36.873611    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" containerID="cri-o://f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8" gracePeriod=30
Jul 31 15:34:36.873786 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:36.873625    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kube-rbac-proxy" containerID="cri-o://6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4" gracePeriod=30
Jul 31 15:34:37.303744 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:37.303708    2561 generic.go:356] "Generic (PLEG): container finished" podID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerID="6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4" exitCode=2
Jul 31 15:34:37.303923 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:37.303781    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" event={"ID":"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5","Type":"ContainerDied","Data":"6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4"}
Jul 31 15:34:37.977584 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:37.977540    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.41:8643/healthz\": dial tcp 10.133.0.41:8643: connect: connection refused"
Jul 31 15:34:37.982943 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:37.982908    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.41:8080: connect: connection refused"
Jul 31 15:34:40.720751 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.720726    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:34:40.765237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.765129    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-cabundle-cert\") pod \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") "
Jul 31 15:34:40.765237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.765186    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-proxy-tls\") pod \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") "
Jul 31 15:34:40.765237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.765235    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fptc7\" (UniqueName: \"kubernetes.io/projected/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-api-access-fptc7\") pod \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") "
Jul 31 15:34:40.765526 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.765299    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kserve-provision-location\") pod \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") "
Jul 31 15:34:40.765526 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.765315    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-rbac-proxy-sar-config\") pod \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\" (UID: \"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5\") "
Jul 31 15:34:40.765708 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.765676    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" (UID: "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:34:40.765831 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.765688    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" (UID: "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:34:40.765877 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.765813    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" (UID: "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:34:40.767486 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.767455    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-api-access-fptc7" (OuterVolumeSpecName: "kube-api-access-fptc7") pod "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" (UID: "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5"). InnerVolumeSpecName "kube-api-access-fptc7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:34:40.767486 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.767471    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" (UID: "dca8f3c1-aa49-4241-9a7d-e9672fb41ac5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:34:40.866337 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.866297    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:34:40.866337 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.866328    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:34:40.866337 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.866341    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:34:40.866337 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.866352    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:34:40.866623 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:40.866360    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fptc7\" (UniqueName: \"kubernetes.io/projected/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5-kube-api-access-fptc7\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:34:41.324764 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.324733    2561 generic.go:356] "Generic (PLEG): container finished" podID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerID="f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8" exitCode=0
Jul 31 15:34:41.324956 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.324785    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" event={"ID":"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5","Type":"ContainerDied","Data":"f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8"}
Jul 31 15:34:41.324956 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.324793    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"
Jul 31 15:34:41.324956 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.324835    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7" event={"ID":"dca8f3c1-aa49-4241-9a7d-e9672fb41ac5","Type":"ContainerDied","Data":"511b2c18882fd3c939e7cbc8ffda3a93222fcaee2a551bc6c86125070bd2cfff"}
Jul 31 15:34:41.324956 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.324864    2561 scope.go:117] "RemoveContainer" containerID="6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4"
Jul 31 15:34:41.333665 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.333644    2561 scope.go:117] "RemoveContainer" containerID="f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8"
Jul 31 15:34:41.341557 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.341541    2561 scope.go:117] "RemoveContainer" containerID="c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326"
Jul 31 15:34:41.347047 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.347024    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"]
Jul 31 15:34:41.349541 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.349524    2561 scope.go:117] "RemoveContainer" containerID="6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4"
Jul 31 15:34:41.349800 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:34:41.349781    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4\": container with ID starting with 6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4 not found: ID does not exist" containerID="6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4"
Jul 31 15:34:41.349852 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.349808    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4"} err="failed to get container status \"6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4\": rpc error: code = NotFound desc = could not find container \"6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4\": container with ID starting with 6ad4a670cbbe211ef62c7c15d60258f07409fdaca61b5012341b9d4cf19342d4 not found: ID does not exist"
Jul 31 15:34:41.349852 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.349824    2561 scope.go:117] "RemoveContainer" containerID="f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8"
Jul 31 15:34:41.350094 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:34:41.350073    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8\": container with ID starting with f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8 not found: ID does not exist" containerID="f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8"
Jul 31 15:34:41.350141 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.350103    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8"} err="failed to get container status \"f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8\": rpc error: code = NotFound desc = could not find container \"f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8\": container with ID starting with f2af928de9ba8dfc81956340e7b691bc28eb03f3b001c6f6eb71e1817e72d2f8 not found: ID does not exist"
Jul 31 15:34:41.350141 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.350124    2561 scope.go:117] "RemoveContainer" containerID="c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326"
Jul 31 15:34:41.350443 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:34:41.350425    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326\": container with ID starting with c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326 not found: ID does not exist" containerID="c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326"
Jul 31 15:34:41.350505 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.350453    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326"} err="failed to get container status \"c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326\": rpc error: code = NotFound desc = could not find container \"c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326\": container with ID starting with c57b9375053389c2b3956703b53c800cc5ac1d7283261ac50785686d62a31326 not found: ID does not exist"
Jul 31 15:34:41.350863 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.350842    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-xgboost-v2-predictor-fcc668bc-wwqb7"]
Jul 31 15:34:41.960899 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:34:41.960871    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" path="/var/lib/kubelet/pods/dca8f3c1-aa49-4241-9a7d-e9672fb41ac5/volumes"
Jul 31 15:35:19.679958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.679919    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"]
Jul 31 15:35:19.680421 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.680263    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container"
Jul 31 15:35:19.680421 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.680276    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container"
Jul 31 15:35:19.680421 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.680289    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="storage-initializer"
Jul 31 15:35:19.680421 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.680294    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="storage-initializer"
Jul 31 15:35:19.680421 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.680309    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kube-rbac-proxy"
Jul 31 15:35:19.680421 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.680315    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kube-rbac-proxy"
Jul 31 15:35:19.680421 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.680365    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kserve-container"
Jul 31 15:35:19.680421 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.680376    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca8f3c1-aa49-4241-9a7d-e9672fb41ac5" containerName="kube-rbac-proxy"
Jul 31 15:35:19.682528 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.682508    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.684997 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.684974    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:35:19.684997 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.684987    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:35:19.685307 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.685292    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:35:19.685393 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.685349    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-predictor-serving-cert\""
Jul 31 15:35:19.685393 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.685371    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-kube-rbac-proxy-sar-config\""
Jul 31 15:35:19.685500 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.685367    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:35:19.686061 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.686044    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:35:19.695575 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.695552    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"]
Jul 31 15:35:19.808790 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.808755    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.809002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.808801    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.809002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.808820    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz9xs\" (UniqueName: \"kubernetes.io/projected/c0fe97c4-531c-409b-8099-0231da4fe290-kube-api-access-fz9xs\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.809002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.808938    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0fe97c4-531c-409b-8099-0231da4fe290-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.809150 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.809027    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.909818 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.909780    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.909978 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.909828    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.909978 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.909873    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.909978 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.909895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz9xs\" (UniqueName: \"kubernetes.io/projected/c0fe97c4-531c-409b-8099-0231da4fe290-kube-api-access-fz9xs\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.909978 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.909948    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0fe97c4-531c-409b-8099-0231da4fe290-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.910265 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:35:19.910023    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-predictor-serving-cert: secret "isvc-paddle-predictor-serving-cert" not found
Jul 31 15:35:19.910265 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:35:19.910124    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls podName:c0fe97c4-531c-409b-8099-0231da4fe290 nodeName:}" failed. No retries permitted until 2026-07-31 15:35:20.410101762 +0000 UTC m=+2816.929525792 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls") pod "isvc-paddle-predictor-6d5c9fd4f6-rwj66" (UID: "c0fe97c4-531c-409b-8099-0231da4fe290") : secret "isvc-paddle-predictor-serving-cert" not found
Jul 31 15:35:19.910393 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.910362    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0fe97c4-531c-409b-8099-0231da4fe290-kserve-provision-location\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.910554 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.910536    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.910591 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.910562    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-cabundle-cert\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:19.922250 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:19.922228    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz9xs\" (UniqueName: \"kubernetes.io/projected/c0fe97c4-531c-409b-8099-0231da4fe290-kube-api-access-fz9xs\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:20.412801 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:20.412759    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:20.415214 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:20.415186    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls\") pod \"isvc-paddle-predictor-6d5c9fd4f6-rwj66\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") " pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:20.593467 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:20.593429    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:20.728181 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:20.728138    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"]
Jul 31 15:35:20.731132 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:35:20.731099    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0fe97c4_531c_409b_8099_0231da4fe290.slice/crio-3e7f2e517a095fdb5eb3e892884cf482655fc80469bcd318aef36af072e57f69 WatchSource:0}: Error finding container 3e7f2e517a095fdb5eb3e892884cf482655fc80469bcd318aef36af072e57f69: Status 404 returned error can't find the container with id 3e7f2e517a095fdb5eb3e892884cf482655fc80469bcd318aef36af072e57f69
Jul 31 15:35:21.493673 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:21.493635    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" event={"ID":"c0fe97c4-531c-409b-8099-0231da4fe290","Type":"ContainerStarted","Data":"502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90"}
Jul 31 15:35:21.493858 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:21.493708    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" event={"ID":"c0fe97c4-531c-409b-8099-0231da4fe290","Type":"ContainerStarted","Data":"3e7f2e517a095fdb5eb3e892884cf482655fc80469bcd318aef36af072e57f69"}
Jul 31 15:35:26.516566 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:26.516529    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0fe97c4-531c-409b-8099-0231da4fe290" containerID="502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90" exitCode=0
Jul 31 15:35:26.517065 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:26.516598    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" event={"ID":"c0fe97c4-531c-409b-8099-0231da4fe290","Type":"ContainerDied","Data":"502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90"}
Jul 31 15:35:26.517846 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:26.517830    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:35:38.575780 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:38.575746    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" event={"ID":"c0fe97c4-531c-409b-8099-0231da4fe290","Type":"ContainerStarted","Data":"0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa"}
Jul 31 15:35:38.576232 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:38.575808    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" event={"ID":"c0fe97c4-531c-409b-8099-0231da4fe290","Type":"ContainerStarted","Data":"7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172"}
Jul 31 15:35:38.576232 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:38.576077    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:38.576232 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:38.576221    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:38.577559 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:38.577532    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:35:38.596821 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:38.596772    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podStartSLOduration=3.346743974 podStartE2EDuration="19.596756608s" totalImagesPullingTime="11.372336399s" totalInitContainerRuntime="4.877676235s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:35:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:35:38.595245398 +0000 UTC m=+2835.114669436" watchObservedRunningTime="2026-07-31 15:35:38.596756608 +0000 UTC m=+2835.116180645"
Jul 31 15:35:39.580515 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:39.580469    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:35:44.585454 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:44.585423    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:35:44.585947 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:44.585922    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:35:54.585901 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:35:54.585864    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:36:04.586105 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:04.586066    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:36:14.586131 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:14.586090    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.42:8080: connect: connection refused"
Jul 31 15:36:24.586866 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:24.586825    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:36:31.348209 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:31.348146    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"]
Jul 31 15:36:31.348698 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:31.348506    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container" containerID="cri-o://7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172" gracePeriod=30
Jul 31 15:36:31.348698 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:31.348520    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kube-rbac-proxy" containerID="cri-o://0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa" gracePeriod=30
Jul 31 15:36:31.806307 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:31.806271    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0fe97c4-531c-409b-8099-0231da4fe290" containerID="0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa" exitCode=2
Jul 31 15:36:31.806488 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:31.806320    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" event={"ID":"c0fe97c4-531c-409b-8099-0231da4fe290","Type":"ContainerDied","Data":"0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa"}
Jul 31 15:36:34.094470 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.094444    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:36:34.174306 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.174209    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls\") pod \"c0fe97c4-531c-409b-8099-0231da4fe290\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") "
Jul 31 15:36:34.174306 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.174245    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0fe97c4-531c-409b-8099-0231da4fe290-kserve-provision-location\") pod \"c0fe97c4-531c-409b-8099-0231da4fe290\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") "
Jul 31 15:36:34.174306 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.174277    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz9xs\" (UniqueName: \"kubernetes.io/projected/c0fe97c4-531c-409b-8099-0231da4fe290-kube-api-access-fz9xs\") pod \"c0fe97c4-531c-409b-8099-0231da4fe290\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") "
Jul 31 15:36:34.174603 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.174405    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-cabundle-cert\") pod \"c0fe97c4-531c-409b-8099-0231da4fe290\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") "
Jul 31 15:36:34.174603 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.174476    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-kube-rbac-proxy-sar-config\") pod \"c0fe97c4-531c-409b-8099-0231da4fe290\" (UID: \"c0fe97c4-531c-409b-8099-0231da4fe290\") "
Jul 31 15:36:34.174812 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.174784    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "c0fe97c4-531c-409b-8099-0231da4fe290" (UID: "c0fe97c4-531c-409b-8099-0231da4fe290"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:36:34.174956 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.174867    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "c0fe97c4-531c-409b-8099-0231da4fe290" (UID: "c0fe97c4-531c-409b-8099-0231da4fe290"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:36:34.176385 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.176363    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "c0fe97c4-531c-409b-8099-0231da4fe290" (UID: "c0fe97c4-531c-409b-8099-0231da4fe290"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:36:34.176493 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.176400    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0fe97c4-531c-409b-8099-0231da4fe290-kube-api-access-fz9xs" (OuterVolumeSpecName: "kube-api-access-fz9xs") pod "c0fe97c4-531c-409b-8099-0231da4fe290" (UID: "c0fe97c4-531c-409b-8099-0231da4fe290"). InnerVolumeSpecName "kube-api-access-fz9xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:36:34.182050 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.182024    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c0fe97c4-531c-409b-8099-0231da4fe290-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "c0fe97c4-531c-409b-8099-0231da4fe290" (UID: "c0fe97c4-531c-409b-8099-0231da4fe290"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:36:34.275193 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.275146    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:36:34.275193 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.275198    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/c0fe97c4-531c-409b-8099-0231da4fe290-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:36:34.275385 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.275212    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/c0fe97c4-531c-409b-8099-0231da4fe290-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:36:34.275385 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.275221    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz9xs\" (UniqueName: \"kubernetes.io/projected/c0fe97c4-531c-409b-8099-0231da4fe290-kube-api-access-fz9xs\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:36:34.275385 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.275230    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/c0fe97c4-531c-409b-8099-0231da4fe290-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:36:34.826893 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.826852    2561 generic.go:356] "Generic (PLEG): container finished" podID="c0fe97c4-531c-409b-8099-0231da4fe290" containerID="7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172" exitCode=0
Jul 31 15:36:34.827101 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.826909    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"
Jul 31 15:36:34.827101 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.826908    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" event={"ID":"c0fe97c4-531c-409b-8099-0231da4fe290","Type":"ContainerDied","Data":"7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172"}
Jul 31 15:36:34.827101 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.826981    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66" event={"ID":"c0fe97c4-531c-409b-8099-0231da4fe290","Type":"ContainerDied","Data":"3e7f2e517a095fdb5eb3e892884cf482655fc80469bcd318aef36af072e57f69"}
Jul 31 15:36:34.827101 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.827018    2561 scope.go:117] "RemoveContainer" containerID="0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa"
Jul 31 15:36:34.835993 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.835976    2561 scope.go:117] "RemoveContainer" containerID="7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172"
Jul 31 15:36:34.843802 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.843784    2561 scope.go:117] "RemoveContainer" containerID="502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90"
Jul 31 15:36:34.850902 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.850877    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"]
Jul 31 15:36:34.852494 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.852472    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-predictor-6d5c9fd4f6-rwj66"]
Jul 31 15:36:34.853263 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.853241    2561 scope.go:117] "RemoveContainer" containerID="0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa"
Jul 31 15:36:34.853593 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:36:34.853576    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa\": container with ID starting with 0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa not found: ID does not exist" containerID="0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa"
Jul 31 15:36:34.853669 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.853602    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa"} err="failed to get container status \"0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa\": rpc error: code = NotFound desc = could not find container \"0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa\": container with ID starting with 0675c70fe73dc69f7894f35ca144b83167d571219bd774f185e9a23f5954b4fa not found: ID does not exist"
Jul 31 15:36:34.853669 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.853621    2561 scope.go:117] "RemoveContainer" containerID="7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172"
Jul 31 15:36:34.853873 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:36:34.853858    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172\": container with ID starting with 7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172 not found: ID does not exist" containerID="7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172"
Jul 31 15:36:34.853917 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.853877    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172"} err="failed to get container status \"7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172\": rpc error: code = NotFound desc = could not find container \"7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172\": container with ID starting with 7b84fc1ed03238ff615826b30ac74fa5ad4a2c330aebbfb3c4f40ce647ca4172 not found: ID does not exist"
Jul 31 15:36:34.853917 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.853892    2561 scope.go:117] "RemoveContainer" containerID="502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90"
Jul 31 15:36:34.854102 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:36:34.854087    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90\": container with ID starting with 502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90 not found: ID does not exist" containerID="502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90"
Jul 31 15:36:34.854164 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:34.854106    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90"} err="failed to get container status \"502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90\": rpc error: code = NotFound desc = could not find container \"502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90\": container with ID starting with 502d7f9f73be09b5c803a90a8d1d8842dfe6a98b1cdff1e3484284478e53de90 not found: ID does not exist"
Jul 31 15:36:35.960624 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:35.960591    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" path="/var/lib/kubelet/pods/c0fe97c4-531c-409b-8099-0231da4fe290/volumes"
Jul 31 15:36:36.512237 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512199    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"]
Jul 31 15:36:36.512630 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512611    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container"
Jul 31 15:36:36.512721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512632    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container"
Jul 31 15:36:36.512721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512667    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="storage-initializer"
Jul 31 15:36:36.512721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512676    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="storage-initializer"
Jul 31 15:36:36.512721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512686    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kube-rbac-proxy"
Jul 31 15:36:36.512721 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512696    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kube-rbac-proxy"
Jul 31 15:36:36.512994 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512768    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kserve-container"
Jul 31 15:36:36.512994 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.512787    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0fe97c4-531c-409b-8099-0231da4fe290" containerName="kube-rbac-proxy"
Jul 31 15:36:36.515872 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.515849    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.518069 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.518047    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-predictor-serving-cert\""
Jul 31 15:36:36.518222 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.518202    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:36:36.518222 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.518201    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:36:36.518368 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.518318    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:36:36.518569 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.518550    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:36:36.518953 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.518934    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-runtime-kube-rbac-proxy-sar-config\""
Jul 31 15:36:36.519049 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.518952    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:36:36.525376 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.525353    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"]
Jul 31 15:36:36.596981 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.596946    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.597139 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.596994    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.597139 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.597064    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.597139 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.597113    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6jb8\" (UniqueName: \"kubernetes.io/projected/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-api-access-z6jb8\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.597288 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.597193    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.698018 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.697982    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.698233 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.698026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.698233 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.698054    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.698233 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.698106    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6jb8\" (UniqueName: \"kubernetes.io/projected/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-api-access-z6jb8\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.698233 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.698199    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.698460 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:36:36.698265    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-runtime-predictor-serving-cert: secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 31 15:36:36.698460 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:36:36.698348    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls podName:78a4a7c8-17f9-490b-b4f8-1a5359a0de39 nodeName:}" failed. No retries permitted until 2026-07-31 15:36:37.198324696 +0000 UTC m=+2893.717748731 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls") pod "isvc-paddle-runtime-predictor-58b99fff76-lkltp" (UID: "78a4a7c8-17f9-490b-b4f8-1a5359a0de39") : secret "isvc-paddle-runtime-predictor-serving-cert" not found
Jul 31 15:36:36.698460 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.698432    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kserve-provision-location\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.698879 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.698858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-cabundle-cert\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.698918 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.698858    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:36.706615 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:36.706588    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6jb8\" (UniqueName: \"kubernetes.io/projected/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-api-access-z6jb8\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:37.201628 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:37.201591    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:37.204147 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:37.204116    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls\") pod \"isvc-paddle-runtime-predictor-58b99fff76-lkltp\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") " pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:37.428249 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:37.428211    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:37.566658 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:37.566583    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"]
Jul 31 15:36:37.568985 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:36:37.568956    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78a4a7c8_17f9_490b_b4f8_1a5359a0de39.slice/crio-6cc0d862eff20d10603c9b439d312c9ecdf65c84305eba7a3f350baae84e7433 WatchSource:0}: Error finding container 6cc0d862eff20d10603c9b439d312c9ecdf65c84305eba7a3f350baae84e7433: Status 404 returned error can't find the container with id 6cc0d862eff20d10603c9b439d312c9ecdf65c84305eba7a3f350baae84e7433
Jul 31 15:36:37.842590 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:37.842556    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" event={"ID":"78a4a7c8-17f9-490b-b4f8-1a5359a0de39","Type":"ContainerStarted","Data":"d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f"}
Jul 31 15:36:37.842765 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:37.842615    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" event={"ID":"78a4a7c8-17f9-490b-b4f8-1a5359a0de39","Type":"ContainerStarted","Data":"6cc0d862eff20d10603c9b439d312c9ecdf65c84305eba7a3f350baae84e7433"}
Jul 31 15:36:42.867407 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:42.867371    2561 generic.go:356] "Generic (PLEG): container finished" podID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerID="d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f" exitCode=0
Jul 31 15:36:42.867828 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:42.867425    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" event={"ID":"78a4a7c8-17f9-490b-b4f8-1a5359a0de39","Type":"ContainerDied","Data":"d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f"}
Jul 31 15:36:43.874816 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:43.874779    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" event={"ID":"78a4a7c8-17f9-490b-b4f8-1a5359a0de39","Type":"ContainerStarted","Data":"68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932"}
Jul 31 15:36:43.875231 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:43.874848    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" event={"ID":"78a4a7c8-17f9-490b-b4f8-1a5359a0de39","Type":"ContainerStarted","Data":"9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816"}
Jul 31 15:36:43.875231 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:43.875039    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:43.896103 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:43.896050    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podStartSLOduration=3.186108129 podStartE2EDuration="7.896030863s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.709922734s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:36:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:36:43.894620455 +0000 UTC m=+2900.414044493" watchObservedRunningTime="2026-07-31 15:36:43.896030863 +0000 UTC m=+2900.415454905"
Jul 31 15:36:44.879461 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:44.879426    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:44.880908 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:44.880878    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:36:45.884483 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:45.884440    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:36:50.889674 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:50.889645    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:36:50.890186 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:36:50.890141    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:37:00.890496 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:00.890456    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:37:10.890862 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:10.890825    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:37:20.890774 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:20.890728    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:37:30.891315 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:30.891285    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:37:38.140974 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:38.140938    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"]
Jul 31 15:37:38.141464 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:38.141262    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container" containerID="cri-o://9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816" gracePeriod=30
Jul 31 15:37:38.141464 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:38.141347    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kube-rbac-proxy" containerID="cri-o://68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932" gracePeriod=30
Jul 31 15:37:39.126148 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:39.126117    2561 generic.go:356] "Generic (PLEG): container finished" podID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerID="68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932" exitCode=2
Jul 31 15:37:39.126335 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:39.126189    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" event={"ID":"78a4a7c8-17f9-490b-b4f8-1a5359a0de39","Type":"ContainerDied","Data":"68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932"}
Jul 31 15:37:40.885265 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:40.884891    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.43:8643/healthz\": dial tcp 10.133.0.43:8643: connect: connection refused"
Jul 31 15:37:40.890568 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:40.890534    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.43:8080: connect: connection refused"
Jul 31 15:37:40.991831 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:40.991806    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:37:41.136564 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.136467    2561 generic.go:356] "Generic (PLEG): container finished" podID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerID="9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816" exitCode=0
Jul 31 15:37:41.136564 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.136520    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"
Jul 31 15:37:41.136799 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.136558    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" event={"ID":"78a4a7c8-17f9-490b-b4f8-1a5359a0de39","Type":"ContainerDied","Data":"9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816"}
Jul 31 15:37:41.136799 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.136667    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp" event={"ID":"78a4a7c8-17f9-490b-b4f8-1a5359a0de39","Type":"ContainerDied","Data":"6cc0d862eff20d10603c9b439d312c9ecdf65c84305eba7a3f350baae84e7433"}
Jul 31 15:37:41.136799 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.136708    2561 scope.go:117] "RemoveContainer" containerID="68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932"
Jul 31 15:37:41.142297 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.142276    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls\") pod \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") "
Jul 31 15:37:41.142394 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.142311    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kserve-provision-location\") pod \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") "
Jul 31 15:37:41.142483 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.142466    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-rbac-proxy-sar-config\") pod \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") "
Jul 31 15:37:41.142533 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.142508    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z6jb8\" (UniqueName: \"kubernetes.io/projected/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-api-access-z6jb8\") pod \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") "
Jul 31 15:37:41.142614 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.142601    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-cabundle-cert\") pod \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\" (UID: \"78a4a7c8-17f9-490b-b4f8-1a5359a0de39\") "
Jul 31 15:37:41.142842 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.142796    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "78a4a7c8-17f9-490b-b4f8-1a5359a0de39" (UID: "78a4a7c8-17f9-490b-b4f8-1a5359a0de39"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:37:41.143074 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.143036    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "78a4a7c8-17f9-490b-b4f8-1a5359a0de39" (UID: "78a4a7c8-17f9-490b-b4f8-1a5359a0de39"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:37:41.144639 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.144613    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "78a4a7c8-17f9-490b-b4f8-1a5359a0de39" (UID: "78a4a7c8-17f9-490b-b4f8-1a5359a0de39"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:37:41.144639 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.144628    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-api-access-z6jb8" (OuterVolumeSpecName: "kube-api-access-z6jb8") pod "78a4a7c8-17f9-490b-b4f8-1a5359a0de39" (UID: "78a4a7c8-17f9-490b-b4f8-1a5359a0de39"). InnerVolumeSpecName "kube-api-access-z6jb8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:37:41.145423 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.145408    2561 scope.go:117] "RemoveContainer" containerID="9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816"
Jul 31 15:37:41.152311 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.152287    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "78a4a7c8-17f9-490b-b4f8-1a5359a0de39" (UID: "78a4a7c8-17f9-490b-b4f8-1a5359a0de39"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:37:41.159447 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.159427    2561 scope.go:117] "RemoveContainer" containerID="d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f"
Jul 31 15:37:41.166628 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.166605    2561 scope.go:117] "RemoveContainer" containerID="68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932"
Jul 31 15:37:41.166876 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:37:41.166858    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932\": container with ID starting with 68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932 not found: ID does not exist" containerID="68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932"
Jul 31 15:37:41.166941 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.166885    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932"} err="failed to get container status \"68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932\": rpc error: code = NotFound desc = could not find container \"68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932\": container with ID starting with 68fe9a428eafb59736926eca35ff23322c2f5c6aff09ee4e1364cc92dc04a932 not found: ID does not exist"
Jul 31 15:37:41.166941 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.166903    2561 scope.go:117] "RemoveContainer" containerID="9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816"
Jul 31 15:37:41.167126 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:37:41.167109    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816\": container with ID starting with 9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816 not found: ID does not exist" containerID="9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816"
Jul 31 15:37:41.167193 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.167129    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816"} err="failed to get container status \"9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816\": rpc error: code = NotFound desc = could not find container \"9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816\": container with ID starting with 9f1d201ad3a66591ad03d99ba010a3a71aa35ff7f413ce094fa9071dd0017816 not found: ID does not exist"
Jul 31 15:37:41.167193 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.167141    2561 scope.go:117] "RemoveContainer" containerID="d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f"
Jul 31 15:37:41.167410 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:37:41.167389    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f\": container with ID starting with d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f not found: ID does not exist" containerID="d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f"
Jul 31 15:37:41.167456 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.167416    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f"} err="failed to get container status \"d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f\": rpc error: code = NotFound desc = could not find container \"d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f\": container with ID starting with d4e928c3a07aa79cc2844b16075a592113da0e8f51553255df6af38c91aa323f not found: ID does not exist"
Jul 31 15:37:41.243477 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.243447    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:37:41.243477 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.243472    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:37:41.243477 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.243481    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:37:41.243685 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.243493    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:37:41.243685 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.243501    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z6jb8\" (UniqueName: \"kubernetes.io/projected/78a4a7c8-17f9-490b-b4f8-1a5359a0de39-kube-api-access-z6jb8\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:37:41.459366 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.459331    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"]
Jul 31 15:37:41.463568 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.463541    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-runtime-predictor-58b99fff76-lkltp"]
Jul 31 15:37:41.960913 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:41.960872    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" path="/var/lib/kubelet/pods/78a4a7c8-17f9-490b-b4f8-1a5359a0de39/volumes"
Jul 31 15:37:43.343970 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.343934    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"]
Jul 31 15:37:43.344392 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.344279    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kube-rbac-proxy"
Jul 31 15:37:43.344392 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.344294    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kube-rbac-proxy"
Jul 31 15:37:43.344392 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.344304    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container"
Jul 31 15:37:43.344392 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.344311    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container"
Jul 31 15:37:43.344392 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.344326    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="storage-initializer"
Jul 31 15:37:43.344392 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.344332    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="storage-initializer"
Jul 31 15:37:43.344392 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.344387    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kube-rbac-proxy"
Jul 31 15:37:43.344659 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.344403    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="78a4a7c8-17f9-490b-b4f8-1a5359a0de39" containerName="kserve-container"
Jul 31 15:37:43.349116 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.349097    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.351422 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.351400    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-predictor-serving-cert\""
Jul 31 15:37:43.351565 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.351516    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:37:43.351565 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.351534    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:37:43.351679 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.351630    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"odh-kserve-custom-ca-bundle\""
Jul 31 15:37:43.351739 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.351709    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-paddle-v2-kserve-kube-rbac-proxy-sar-config\""
Jul 31 15:37:43.352172 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.352141    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:37:43.352222 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.352201    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"seaweedfs-s3-creds\""
Jul 31 15:37:43.357362 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.357341    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"]
Jul 31 15:37:43.461480 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.461441    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aadaf3b9-78cc-4801-b839-78f88409b1b6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.461667 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.461489    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmhmk\" (UniqueName: \"kubernetes.io/projected/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-api-access-fmhmk\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.461667 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.461523    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.461667 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.461584    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.461824 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.461720    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.562637 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.562595    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aadaf3b9-78cc-4801-b839-78f88409b1b6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.562637 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.562641    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmhmk\" (UniqueName: \"kubernetes.io/projected/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-api-access-fmhmk\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.562923 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.562663    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.562923 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.562684    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.562923 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.562752    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.562923 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:37:43.562805    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-serving-cert: secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 31 15:37:43.562923 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:37:43.562901    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls podName:aadaf3b9-78cc-4801-b839-78f88409b1b6 nodeName:}" failed. No retries permitted until 2026-07-31 15:37:44.062879863 +0000 UTC m=+2960.582303892 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls") pod "isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" (UID: "aadaf3b9-78cc-4801-b839-78f88409b1b6") : secret "isvc-paddle-v2-kserve-predictor-serving-cert" not found
Jul 31 15:37:43.563231 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.563091    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aadaf3b9-78cc-4801-b839-78f88409b1b6-kserve-provision-location\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.563562 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.563541    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-rbac-proxy-sar-config\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.563606 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.563547    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-cabundle-cert\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:43.574990 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:43.574955    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmhmk\" (UniqueName: \"kubernetes.io/projected/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-api-access-fmhmk\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:44.068132 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:44.068096    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:44.070606 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:44.070581    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls\") pod \"isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") " pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:44.259890 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:44.259861    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:44.392817 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:44.392790    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"]
Jul 31 15:37:44.395345 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:37:44.395315    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaadaf3b9_78cc_4801_b839_78f88409b1b6.slice/crio-10b6809730d7bfbf550e4c4c0b3c7b01d1d5ae267283655800fff91cb9af1418 WatchSource:0}: Error finding container 10b6809730d7bfbf550e4c4c0b3c7b01d1d5ae267283655800fff91cb9af1418: Status 404 returned error can't find the container with id 10b6809730d7bfbf550e4c4c0b3c7b01d1d5ae267283655800fff91cb9af1418
Jul 31 15:37:45.156559 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:45.156523    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" event={"ID":"aadaf3b9-78cc-4801-b839-78f88409b1b6","Type":"ContainerStarted","Data":"ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf"}
Jul 31 15:37:45.156747 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:45.156585    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" event={"ID":"aadaf3b9-78cc-4801-b839-78f88409b1b6","Type":"ContainerStarted","Data":"10b6809730d7bfbf550e4c4c0b3c7b01d1d5ae267283655800fff91cb9af1418"}
Jul 31 15:37:49.176481 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:49.176445    2561 generic.go:356] "Generic (PLEG): container finished" podID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerID="ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf" exitCode=0
Jul 31 15:37:49.176939 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:49.176511    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" event={"ID":"aadaf3b9-78cc-4801-b839-78f88409b1b6","Type":"ContainerDied","Data":"ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf"}
Jul 31 15:37:50.183385 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:50.183352    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" event={"ID":"aadaf3b9-78cc-4801-b839-78f88409b1b6","Type":"ContainerStarted","Data":"d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe"}
Jul 31 15:37:50.183779 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:50.183418    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" event={"ID":"aadaf3b9-78cc-4801-b839-78f88409b1b6","Type":"ContainerStarted","Data":"7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8"}
Jul 31 15:37:50.183779 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:50.183616    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:50.204748 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:50.204702    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podStartSLOduration=2.686747566 podStartE2EDuration="7.20468631s" totalImagesPullingTime="0s" totalInitContainerRuntime="4.517938744s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:37:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:37:50.20330534 +0000 UTC m=+2966.722729379" watchObservedRunningTime="2026-07-31 15:37:50.20468631 +0000 UTC m=+2966.724110346"
Jul 31 15:37:51.188070 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:51.188040    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:51.189635 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:51.189603    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:37:52.195302 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:52.195242    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:37:57.198387 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:57.198350    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:37:57.199060 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:37:57.199033    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:38:07.199002 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:07.198962    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:38:17.199500 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:17.199444    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:38:24.150119 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:24.150081    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:38:24.158134 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:24.158101    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:38:27.199646 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:27.199602    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:38:37.200330 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:37.200300    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:38:45.253054 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:45.253013    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"]
Jul 31 15:38:45.253652 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:45.253300    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kube-rbac-proxy" containerID="cri-o://d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe" gracePeriod=30
Jul 31 15:38:45.253652 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:45.253292    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container" containerID="cri-o://7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8" gracePeriod=30
Jul 31 15:38:45.425639 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:45.425609    2561 generic.go:356] "Generic (PLEG): container finished" podID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerID="d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe" exitCode=2
Jul 31 15:38:45.425772 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:45.425653    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" event={"ID":"aadaf3b9-78cc-4801-b839-78f88409b1b6","Type":"ContainerDied","Data":"d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe"}
Jul 31 15:38:47.194505 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.194459    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.133.0.44:8643/healthz\": dial tcp 10.133.0.44:8643: connect: connection refused"
Jul 31 15:38:47.199994 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.199941    2561 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.133.0.44:8080: connect: connection refused"
Jul 31 15:38:47.899664 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.899636    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:38:47.924762 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.924728    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-cabundle-cert\") pod \"aadaf3b9-78cc-4801-b839-78f88409b1b6\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") "
Jul 31 15:38:47.924958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.924773    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls\") pod \"aadaf3b9-78cc-4801-b839-78f88409b1b6\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") "
Jul 31 15:38:47.924958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.924815    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fmhmk\" (UniqueName: \"kubernetes.io/projected/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-api-access-fmhmk\") pod \"aadaf3b9-78cc-4801-b839-78f88409b1b6\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") "
Jul 31 15:38:47.924958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.924844    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-rbac-proxy-sar-config\") pod \"aadaf3b9-78cc-4801-b839-78f88409b1b6\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") "
Jul 31 15:38:47.924958 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.924868    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aadaf3b9-78cc-4801-b839-78f88409b1b6-kserve-provision-location\") pod \"aadaf3b9-78cc-4801-b839-78f88409b1b6\" (UID: \"aadaf3b9-78cc-4801-b839-78f88409b1b6\") "
Jul 31 15:38:47.925230 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.925176    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-cabundle-cert" (OuterVolumeSpecName: "cabundle-cert") pod "aadaf3b9-78cc-4801-b839-78f88409b1b6" (UID: "aadaf3b9-78cc-4801-b839-78f88409b1b6"). InnerVolumeSpecName "cabundle-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:38:47.925328 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.925295    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "aadaf3b9-78cc-4801-b839-78f88409b1b6" (UID: "aadaf3b9-78cc-4801-b839-78f88409b1b6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:38:47.927109 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.927077    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "aadaf3b9-78cc-4801-b839-78f88409b1b6" (UID: "aadaf3b9-78cc-4801-b839-78f88409b1b6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:38:47.927238 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.927083    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-api-access-fmhmk" (OuterVolumeSpecName: "kube-api-access-fmhmk") pod "aadaf3b9-78cc-4801-b839-78f88409b1b6" (UID: "aadaf3b9-78cc-4801-b839-78f88409b1b6"). InnerVolumeSpecName "kube-api-access-fmhmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:38:47.934945 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:47.934915    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/aadaf3b9-78cc-4801-b839-78f88409b1b6-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "aadaf3b9-78cc-4801-b839-78f88409b1b6" (UID: "aadaf3b9-78cc-4801-b839-78f88409b1b6"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:38:48.026043 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.026006    2561 reconciler_common.go:299] "Volume detached for volume \"cabundle-cert\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-cabundle-cert\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:38:48.026043 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.026043    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/aadaf3b9-78cc-4801-b839-78f88409b1b6-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:38:48.026256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.026055    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fmhmk\" (UniqueName: \"kubernetes.io/projected/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-api-access-fmhmk\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:38:48.026256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.026068    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/aadaf3b9-78cc-4801-b839-78f88409b1b6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:38:48.026256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.026082    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/aadaf3b9-78cc-4801-b839-78f88409b1b6-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:38:48.439259 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.439224    2561 generic.go:356] "Generic (PLEG): container finished" podID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerID="7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8" exitCode=0
Jul 31 15:38:48.439643 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.439311    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"
Jul 31 15:38:48.439643 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.439314    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" event={"ID":"aadaf3b9-78cc-4801-b839-78f88409b1b6","Type":"ContainerDied","Data":"7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8"}
Jul 31 15:38:48.439643 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.439426    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x" event={"ID":"aadaf3b9-78cc-4801-b839-78f88409b1b6","Type":"ContainerDied","Data":"10b6809730d7bfbf550e4c4c0b3c7b01d1d5ae267283655800fff91cb9af1418"}
Jul 31 15:38:48.439643 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.439465    2561 scope.go:117] "RemoveContainer" containerID="d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe"
Jul 31 15:38:48.447891 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.447867    2561 scope.go:117] "RemoveContainer" containerID="7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8"
Jul 31 15:38:48.455277 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.455258    2561 scope.go:117] "RemoveContainer" containerID="ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf"
Jul 31 15:38:48.459330 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.459310    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"]
Jul 31 15:38:48.462757 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.462742    2561 scope.go:117] "RemoveContainer" containerID="d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe"
Jul 31 15:38:48.463025 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:38:48.463003    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe\": container with ID starting with d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe not found: ID does not exist" containerID="d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe"
Jul 31 15:38:48.463115 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.463030    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe"} err="failed to get container status \"d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe\": rpc error: code = NotFound desc = could not find container \"d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe\": container with ID starting with d0b14d705fa6b7ccca516f9307efddd92abf704992eae25e4ef8bd98e6414ffe not found: ID does not exist"
Jul 31 15:38:48.463115 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.463048    2561 scope.go:117] "RemoveContainer" containerID="7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8"
Jul 31 15:38:48.463386 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:38:48.463365    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8\": container with ID starting with 7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8 not found: ID does not exist" containerID="7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8"
Jul 31 15:38:48.463477 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.463390    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8"} err="failed to get container status \"7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8\": rpc error: code = NotFound desc = could not find container \"7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8\": container with ID starting with 7000b0db50572af673400d241091a794720fd450e489ecdd0f56dbd106386fc8 not found: ID does not exist"
Jul 31 15:38:48.463477 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.463405    2561 scope.go:117] "RemoveContainer" containerID="ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf"
Jul 31 15:38:48.463689 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:38:48.463665    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf\": container with ID starting with ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf not found: ID does not exist" containerID="ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf"
Jul 31 15:38:48.463739 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.463696    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf"} err="failed to get container status \"ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf\": rpc error: code = NotFound desc = could not find container \"ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf\": container with ID starting with ab52ea066a6a81c3163603debf3bf9af782241408a7b8d20e178f519bcebb6bf not found: ID does not exist"
Jul 31 15:38:48.466238 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:48.466220    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-paddle-v2-kserve-predictor-64d4c74b4d-wjw8x"]
Jul 31 15:38:49.962545 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:38:49.962508    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" path="/var/lib/kubelet/pods/aadaf3b9-78cc-4801-b839-78f88409b1b6/volumes"
Jul 31 15:39:15.896535 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896457    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"]
Jul 31 15:39:15.897001 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896779    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container"
Jul 31 15:39:15.897001 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896790    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container"
Jul 31 15:39:15.897001 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896801    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kube-rbac-proxy"
Jul 31 15:39:15.897001 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896807    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kube-rbac-proxy"
Jul 31 15:39:15.897001 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896814    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="storage-initializer"
Jul 31 15:39:15.897001 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896821    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="storage-initializer"
Jul 31 15:39:15.897001 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896891    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kserve-container"
Jul 31 15:39:15.897001 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.896902    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="aadaf3b9-78cc-4801-b839-78f88409b1b6" containerName="kube-rbac-proxy"
Jul 31 15:39:15.899910 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.899893    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:15.903147 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.903123    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 31 15:39:15.903334 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.903313    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:39:15.903489 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.903470    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-kube-rbac-proxy-sar-config\""
Jul 31 15:39:15.904117 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.904101    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:39:15.907337 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.907322    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:39:15.908883 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.908866    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-custom-fail-predictor-serving-cert\""
Jul 31 15:39:15.920013 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:15.919991    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"]
Jul 31 15:39:16.068765 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.068736    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dbvjv\" (UniqueName: \"kubernetes.io/projected/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-api-access-dbvjv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.068961 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.068783    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.068961 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.068838    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.068961 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.068865    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.170058 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.169963    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.170058 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.170026    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dbvjv\" (UniqueName: \"kubernetes.io/projected/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-api-access-dbvjv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.170353 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.170081    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.170353 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.170135    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.170470 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.170448    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.170840 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.170819    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.172547 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.172530    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-proxy-tls\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.180439 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.180413    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dbvjv\" (UniqueName: \"kubernetes.io/projected/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-api-access-dbvjv\") pod \"isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.209803 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.209765    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:16.368104 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.368065    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"]
Jul 31 15:39:16.376533 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:39:16.376501    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8848bc_8d03_4377_a4e0_25796f5e8eaf.slice/crio-e3ff905d84740d49d6f766dac4a2d204d97ab149876782232a1f9665135e93ca WatchSource:0}: Error finding container e3ff905d84740d49d6f766dac4a2d204d97ab149876782232a1f9665135e93ca: Status 404 returned error can't find the container with id e3ff905d84740d49d6f766dac4a2d204d97ab149876782232a1f9665135e93ca
Jul 31 15:39:16.555817 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.555764    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w" event={"ID":"4b8848bc-8d03-4377-a4e0-25796f5e8eaf","Type":"ContainerStarted","Data":"85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844"}
Jul 31 15:39:16.556023 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:16.555854    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w" event={"ID":"4b8848bc-8d03-4377-a4e0-25796f5e8eaf","Type":"ContainerStarted","Data":"e3ff905d84740d49d6f766dac4a2d204d97ab149876782232a1f9665135e93ca"}
Jul 31 15:39:23.590163 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:23.590134    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w_4b8848bc-8d03-4377-a4e0-25796f5e8eaf/storage-initializer/0.log"
Jul 31 15:39:23.590565 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:23.590192    2561 generic.go:356] "Generic (PLEG): container finished" podID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerID="85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844" exitCode=1
Jul 31 15:39:23.590565 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:23.590273    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w" event={"ID":"4b8848bc-8d03-4377-a4e0-25796f5e8eaf","Type":"ContainerDied","Data":"85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844"}
Jul 31 15:39:24.596461 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:24.596383    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w_4b8848bc-8d03-4377-a4e0-25796f5e8eaf/storage-initializer/0.log"
Jul 31 15:39:24.596461 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:24.596441    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w" event={"ID":"4b8848bc-8d03-4377-a4e0-25796f5e8eaf","Type":"ContainerStarted","Data":"b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401"}
Jul 31 15:39:25.773350 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:25.773312    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"]
Jul 31 15:39:25.773781 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:25.773523    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w" podUID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerName="storage-initializer" containerID="cri-o://b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401" gracePeriod=30
Jul 31 15:39:28.319899 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.319872    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w_4b8848bc-8d03-4377-a4e0-25796f5e8eaf/storage-initializer/1.log"
Jul 31 15:39:28.320306 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.320291    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w_4b8848bc-8d03-4377-a4e0-25796f5e8eaf/storage-initializer/0.log"
Jul 31 15:39:28.322505 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.322492    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:28.478939 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.478840    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dbvjv\" (UniqueName: \"kubernetes.io/projected/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-api-access-dbvjv\") pod \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") "
Jul 31 15:39:28.478939 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.478913    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-rbac-proxy-sar-config\") pod \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") "
Jul 31 15:39:28.479202 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.478967    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kserve-provision-location\") pod \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") "
Jul 31 15:39:28.479202 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.479040    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-proxy-tls\") pod \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\" (UID: \"4b8848bc-8d03-4377-a4e0-25796f5e8eaf\") "
Jul 31 15:39:28.479306 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.479270    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "4b8848bc-8d03-4377-a4e0-25796f5e8eaf" (UID: "4b8848bc-8d03-4377-a4e0-25796f5e8eaf"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:39:28.479344 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.479280    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "4b8848bc-8d03-4377-a4e0-25796f5e8eaf" (UID: "4b8848bc-8d03-4377-a4e0-25796f5e8eaf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:39:28.480977 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.480953    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-api-access-dbvjv" (OuterVolumeSpecName: "kube-api-access-dbvjv") pod "4b8848bc-8d03-4377-a4e0-25796f5e8eaf" (UID: "4b8848bc-8d03-4377-a4e0-25796f5e8eaf"). InnerVolumeSpecName "kube-api-access-dbvjv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:39:28.481089 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.481075    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "4b8848bc-8d03-4377-a4e0-25796f5e8eaf" (UID: "4b8848bc-8d03-4377-a4e0-25796f5e8eaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:39:28.579695 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.579657    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:39:28.579695 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.579691    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dbvjv\" (UniqueName: \"kubernetes.io/projected/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-api-access-dbvjv\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:39:28.579695 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.579702    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:39:28.579939 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.579711    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/4b8848bc-8d03-4377-a4e0-25796f5e8eaf-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:39:28.614340 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.614312    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w_4b8848bc-8d03-4377-a4e0-25796f5e8eaf/storage-initializer/1.log"
Jul 31 15:39:28.614677 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.614663    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w_4b8848bc-8d03-4377-a4e0-25796f5e8eaf/storage-initializer/0.log"
Jul 31 15:39:28.614725 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.614698    2561 generic.go:356] "Generic (PLEG): container finished" podID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerID="b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401" exitCode=1
Jul 31 15:39:28.614781 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.614764    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"
Jul 31 15:39:28.614822 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.614784    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w" event={"ID":"4b8848bc-8d03-4377-a4e0-25796f5e8eaf","Type":"ContainerDied","Data":"b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401"}
Jul 31 15:39:28.614886 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.614874    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w" event={"ID":"4b8848bc-8d03-4377-a4e0-25796f5e8eaf","Type":"ContainerDied","Data":"e3ff905d84740d49d6f766dac4a2d204d97ab149876782232a1f9665135e93ca"}
Jul 31 15:39:28.614921 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.614904    2561 scope.go:117] "RemoveContainer" containerID="b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401"
Jul 31 15:39:28.623732 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.623709    2561 scope.go:117] "RemoveContainer" containerID="85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844"
Jul 31 15:39:28.631544 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.631527    2561 scope.go:117] "RemoveContainer" containerID="b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401"
Jul 31 15:39:28.631821 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:39:28.631803    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401\": container with ID starting with b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401 not found: ID does not exist" containerID="b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401"
Jul 31 15:39:28.631871 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.631831    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401"} err="failed to get container status \"b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401\": rpc error: code = NotFound desc = could not find container \"b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401\": container with ID starting with b68ee52f6933ad4a4022a187bdb417aa53120548633ca670f129390b7ae0d401 not found: ID does not exist"
Jul 31 15:39:28.631871 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.631851    2561 scope.go:117] "RemoveContainer" containerID="85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844"
Jul 31 15:39:28.632116 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:39:28.632098    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844\": container with ID starting with 85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844 not found: ID does not exist" containerID="85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844"
Jul 31 15:39:28.632189 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.632125    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844"} err="failed to get container status \"85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844\": rpc error: code = NotFound desc = could not find container \"85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844\": container with ID starting with 85f765338bfccd8dfd75b98ece840eb91d9da782326093e9c6125a70177ad844 not found: ID does not exist"
Jul 31 15:39:28.666467 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.666426    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"]
Jul 31 15:39:28.671483 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:28.671452    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-custom-fail-predictor-56cd97f966-lpz7w"]
Jul 31 15:39:29.961024 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:39:29.960987    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" path="/var/lib/kubelet/pods/4b8848bc-8d03-4377-a4e0-25796f5e8eaf/volumes"
Jul 31 15:40:47.944600 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.944514    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"]
Jul 31 15:40:47.945128 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.944828    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerName="storage-initializer"
Jul 31 15:40:47.945128 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.944839    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerName="storage-initializer"
Jul 31 15:40:47.945128 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.944910    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerName="storage-initializer"
Jul 31 15:40:47.945128 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.944919    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerName="storage-initializer"
Jul 31 15:40:47.945128 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.944975    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerName="storage-initializer"
Jul 31 15:40:47.945128 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.944982    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b8848bc-8d03-4377-a4e0-25796f5e8eaf" containerName="storage-initializer"
Jul 31 15:40:47.947985 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.947963    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:47.950325 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.950300    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 31 15:40:47.950464 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.950300    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 31 15:40:47.950464 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.950302    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"storage-config\""
Jul 31 15:40:47.950582 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.950330    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-kube-rbac-proxy-sar-config\""
Jul 31 15:40:47.950582 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.950361    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-dd67w\""
Jul 31 15:40:47.950859 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.950845    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert\""
Jul 31 15:40:47.963172 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:47.963126    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"]
Jul 31 15:40:48.020515 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.020476    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.020515 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.020516    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.020735 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.020540    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.020735 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.020621    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsdmh\" (UniqueName: \"kubernetes.io/projected/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-api-access-zsdmh\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.121379 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.121342    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.121379 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.121383    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.121628 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.121413    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.121628 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.121459    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsdmh\" (UniqueName: \"kubernetes.io/projected/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-api-access-zsdmh\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.121628 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:40:48.121514    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 31 15:40:48.121628 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:40:48.121592    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls podName:9015a7f7-76cc-4b29-96f8-4ebfa968620e nodeName:}" failed. No retries permitted until 2026-07-31 15:40:48.62157111 +0000 UTC m=+3145.140995131 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" (UID: "9015a7f7-76cc-4b29-96f8-4ebfa968620e") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 31 15:40:48.121911 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.121889    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kserve-provision-location\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.122072 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.122051    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-rbac-proxy-sar-config\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.130265 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.130238    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsdmh\" (UniqueName: \"kubernetes.io/projected/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-api-access-zsdmh\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.625519 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:48.625473    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:48.625709 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:40:48.625613    2561 secret.go:189] Couldn't get secret kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert: secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 31 15:40:48.625709 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:40:48.625679    2561 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls podName:9015a7f7-76cc-4b29-96f8-4ebfa968620e nodeName:}" failed. No retries permitted until 2026-07-31 15:40:49.625662889 +0000 UTC m=+3146.145086906 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls") pod "isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" (UID: "9015a7f7-76cc-4b29-96f8-4ebfa968620e") : secret "isvc-sklearn-s3-tls-serving-fail-predictor-serving-cert" not found
Jul 31 15:40:49.634238 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:49.634200    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:49.636730 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:49.636704    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls\") pod \"isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") " pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:49.760122 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:49.760067    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:49.896965 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:49.896938    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"]
Jul 31 15:40:49.899620 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:40:49.899590    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9015a7f7_76cc_4b29_96f8_4ebfa968620e.slice/crio-7a739dc9b7c4ed66945c63726798b30351b9433cee5dcc2ea69cf62e18d9a063 WatchSource:0}: Error finding container 7a739dc9b7c4ed66945c63726798b30351b9433cee5dcc2ea69cf62e18d9a063: Status 404 returned error can't find the container with id 7a739dc9b7c4ed66945c63726798b30351b9433cee5dcc2ea69cf62e18d9a063
Jul 31 15:40:49.975796 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:49.975743    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" event={"ID":"9015a7f7-76cc-4b29-96f8-4ebfa968620e","Type":"ContainerStarted","Data":"a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea"}
Jul 31 15:40:49.975978 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:49.975846    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" event={"ID":"9015a7f7-76cc-4b29-96f8-4ebfa968620e","Type":"ContainerStarted","Data":"7a739dc9b7c4ed66945c63726798b30351b9433cee5dcc2ea69cf62e18d9a063"}
Jul 31 15:40:52.991723 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:52.991699    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_9015a7f7-76cc-4b29-96f8-4ebfa968620e/storage-initializer/0.log"
Jul 31 15:40:52.992090 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:52.991736    2561 generic.go:356] "Generic (PLEG): container finished" podID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" containerID="a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea" exitCode=1
Jul 31 15:40:52.992090 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:52.991810    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" event={"ID":"9015a7f7-76cc-4b29-96f8-4ebfa968620e","Type":"ContainerDied","Data":"a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea"}
Jul 31 15:40:53.997346 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:53.997313    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_9015a7f7-76cc-4b29-96f8-4ebfa968620e/storage-initializer/0.log"
Jul 31 15:40:53.997724 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:53.997402    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" event={"ID":"9015a7f7-76cc-4b29-96f8-4ebfa968620e","Type":"ContainerStarted","Data":"002b862a2b4d638299f02b6fcadeddc9424de25501985aaf43061d9ce6e132b4"}
Jul 31 15:40:57.010648 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:57.010618    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_9015a7f7-76cc-4b29-96f8-4ebfa968620e/storage-initializer/1.log"
Jul 31 15:40:57.011089 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:57.010992    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_9015a7f7-76cc-4b29-96f8-4ebfa968620e/storage-initializer/0.log"
Jul 31 15:40:57.011089 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:57.011020    2561 generic.go:356] "Generic (PLEG): container finished" podID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" containerID="002b862a2b4d638299f02b6fcadeddc9424de25501985aaf43061d9ce6e132b4" exitCode=1
Jul 31 15:40:57.011089 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:57.011077    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" event={"ID":"9015a7f7-76cc-4b29-96f8-4ebfa968620e","Type":"ContainerDied","Data":"002b862a2b4d638299f02b6fcadeddc9424de25501985aaf43061d9ce6e132b4"}
Jul 31 15:40:57.011256 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:57.011129    2561 scope.go:117] "RemoveContainer" containerID="a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea"
Jul 31 15:40:57.011519 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:57.011499    2561 scope.go:117] "RemoveContainer" containerID="a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea"
Jul 31 15:40:57.024464 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:40:57.024440    2561 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"storage-initializer\" with CrashLoopBackOff: \"back-off 10s restarting failed container=storage-initializer pod=isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_kserve-ci-e2e-test(9015a7f7-76cc-4b29-96f8-4ebfa968620e)\"" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" podUID="9015a7f7-76cc-4b29-96f8-4ebfa968620e"
Jul 31 15:40:57.024692 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:40:57.024663    2561 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_kserve-ci-e2e-test_9015a7f7-76cc-4b29-96f8-4ebfa968620e_0 in pod sandbox 7a739dc9b7c4ed66945c63726798b30351b9433cee5dcc2ea69cf62e18d9a063 from index: no such id: 'a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea'" containerID="a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea"
Jul 31 15:40:57.024746 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:57.024704    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea"} err="rpc error: code = Unknown desc = failed to delete container k8s_storage-initializer_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_kserve-ci-e2e-test_9015a7f7-76cc-4b29-96f8-4ebfa968620e_0 in pod sandbox 7a739dc9b7c4ed66945c63726798b30351b9433cee5dcc2ea69cf62e18d9a063 from index: no such id: 'a393d69a1dd69a1f55d08daaeb5d4175416715e574e586be7901cee927b7d3ea'"
Jul 31 15:40:57.938353 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:57.938317    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"]
Jul 31 15:40:58.016455 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.016428    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_9015a7f7-76cc-4b29-96f8-4ebfa968620e/storage-initializer/1.log"
Jul 31 15:40:58.142434 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.142413    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_9015a7f7-76cc-4b29-96f8-4ebfa968620e/storage-initializer/1.log"
Jul 31 15:40:58.145899 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.145882    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:58.208622 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.208534    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kserve-provision-location\") pod \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") "
Jul 31 15:40:58.208775 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.208625    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsdmh\" (UniqueName: \"kubernetes.io/projected/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-api-access-zsdmh\") pod \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") "
Jul 31 15:40:58.208775 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.208657    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-rbac-proxy-sar-config\") pod \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") "
Jul 31 15:40:58.208775 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.208694    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls\") pod \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\" (UID: \"9015a7f7-76cc-4b29-96f8-4ebfa968620e\") "
Jul 31 15:40:58.208881 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.208822    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kserve-provision-location" (OuterVolumeSpecName: "kserve-provision-location") pod "9015a7f7-76cc-4b29-96f8-4ebfa968620e" (UID: "9015a7f7-76cc-4b29-96f8-4ebfa968620e"). InnerVolumeSpecName "kserve-provision-location". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:40:58.208995 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.208978    2561 reconciler_common.go:299] "Volume detached for volume \"kserve-provision-location\" (UniqueName: \"kubernetes.io/empty-dir/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kserve-provision-location\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:40:58.209045 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.209028    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9015a7f7-76cc-4b29-96f8-4ebfa968620e" (UID: "9015a7f7-76cc-4b29-96f8-4ebfa968620e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 31 15:40:58.210801 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.210777    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9015a7f7-76cc-4b29-96f8-4ebfa968620e" (UID: "9015a7f7-76cc-4b29-96f8-4ebfa968620e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 31 15:40:58.210801 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.210781    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-api-access-zsdmh" (OuterVolumeSpecName: "kube-api-access-zsdmh") pod "9015a7f7-76cc-4b29-96f8-4ebfa968620e" (UID: "9015a7f7-76cc-4b29-96f8-4ebfa968620e"). InnerVolumeSpecName "kube-api-access-zsdmh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:40:58.310401 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.310364    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsdmh\" (UniqueName: \"kubernetes.io/projected/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-api-access-zsdmh\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:40:58.310401 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.310394    2561 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9015a7f7-76cc-4b29-96f8-4ebfa968620e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:40:58.310401 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:58.310404    2561 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9015a7f7-76cc-4b29-96f8-4ebfa968620e-proxy-tls\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:40:59.021998 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:59.021965    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve-ci-e2e-test_isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p_9015a7f7-76cc-4b29-96f8-4ebfa968620e/storage-initializer/1.log"
Jul 31 15:40:59.022467 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:59.022038    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p" event={"ID":"9015a7f7-76cc-4b29-96f8-4ebfa968620e","Type":"ContainerDied","Data":"7a739dc9b7c4ed66945c63726798b30351b9433cee5dcc2ea69cf62e18d9a063"}
Jul 31 15:40:59.022467 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:59.022079    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"
Jul 31 15:40:59.022467 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:59.022102    2561 scope.go:117] "RemoveContainer" containerID="002b862a2b4d638299f02b6fcadeddc9424de25501985aaf43061d9ce6e132b4"
Jul 31 15:40:59.063630 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:59.063599    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"]
Jul 31 15:40:59.072116 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:59.072083    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/isvc-sklearn-s3-tls-serving-fail-predictor-6bc8d5c5c9-jbd9p"]
Jul 31 15:40:59.961363 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:40:59.961322    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" path="/var/lib/kubelet/pods/9015a7f7-76cc-4b29-96f8-4ebfa968620e/volumes"
Jul 31 15:42:13.910656 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.910614    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nsj8w/must-gather-xrn4l"]
Jul 31 15:42:13.911192 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.910943    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" containerName="storage-initializer"
Jul 31 15:42:13.911192 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.910955    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" containerName="storage-initializer"
Jul 31 15:42:13.911192 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.910962    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" containerName="storage-initializer"
Jul 31 15:42:13.911192 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.910968    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" containerName="storage-initializer"
Jul 31 15:42:13.911192 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.911031    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" containerName="storage-initializer"
Jul 31 15:42:13.911192 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.911041    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="9015a7f7-76cc-4b29-96f8-4ebfa968620e" containerName="storage-initializer"
Jul 31 15:42:13.914031 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.914015    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:13.916145 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.916121    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsj8w\"/\"kube-root-ca.crt\""
Jul 31 15:42:13.916276 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.916213    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nsj8w\"/\"openshift-service-ca.crt\""
Jul 31 15:42:13.924145 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:13.924116    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsj8w/must-gather-xrn4l"]
Jul 31 15:42:14.058480 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.058439    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d3903e-9d74-4df2-af63-6a897168f8c5-must-gather-output\") pod \"must-gather-xrn4l\" (UID: \"19d3903e-9d74-4df2-af63-6a897168f8c5\") " pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:14.058665 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.058488    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-st7nk\" (UniqueName: \"kubernetes.io/projected/19d3903e-9d74-4df2-af63-6a897168f8c5-kube-api-access-st7nk\") pod \"must-gather-xrn4l\" (UID: \"19d3903e-9d74-4df2-af63-6a897168f8c5\") " pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:14.159757 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.159713    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d3903e-9d74-4df2-af63-6a897168f8c5-must-gather-output\") pod \"must-gather-xrn4l\" (UID: \"19d3903e-9d74-4df2-af63-6a897168f8c5\") " pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:14.159938 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.159766    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-st7nk\" (UniqueName: \"kubernetes.io/projected/19d3903e-9d74-4df2-af63-6a897168f8c5-kube-api-access-st7nk\") pod \"must-gather-xrn4l\" (UID: \"19d3903e-9d74-4df2-af63-6a897168f8c5\") " pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:14.160101 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.160079    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d3903e-9d74-4df2-af63-6a897168f8c5-must-gather-output\") pod \"must-gather-xrn4l\" (UID: \"19d3903e-9d74-4df2-af63-6a897168f8c5\") " pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:14.169088 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.169011    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-st7nk\" (UniqueName: \"kubernetes.io/projected/19d3903e-9d74-4df2-af63-6a897168f8c5-kube-api-access-st7nk\") pod \"must-gather-xrn4l\" (UID: \"19d3903e-9d74-4df2-af63-6a897168f8c5\") " pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:14.229315 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.229270    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:14.366062 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.366021    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nsj8w/must-gather-xrn4l"]
Jul 31 15:42:14.369356 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:42:14.369330    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod19d3903e_9d74_4df2_af63_6a897168f8c5.slice/crio-193c1c79204c47a2f11b3a8cdfbb5b82e62398998568b77d8e1a2bb8ac183fba WatchSource:0}: Error finding container 193c1c79204c47a2f11b3a8cdfbb5b82e62398998568b77d8e1a2bb8ac183fba: Status 404 returned error can't find the container with id 193c1c79204c47a2f11b3a8cdfbb5b82e62398998568b77d8e1a2bb8ac183fba
Jul 31 15:42:14.371565 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:14.371541    2561 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 31 15:42:15.343295 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:15.343053    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsj8w/must-gather-xrn4l" event={"ID":"19d3903e-9d74-4df2-af63-6a897168f8c5","Type":"ContainerStarted","Data":"193c1c79204c47a2f11b3a8cdfbb5b82e62398998568b77d8e1a2bb8ac183fba"}
Jul 31 15:42:21.375169 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:21.375110    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsj8w/must-gather-xrn4l" event={"ID":"19d3903e-9d74-4df2-af63-6a897168f8c5","Type":"ContainerStarted","Data":"c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38"}
Jul 31 15:42:21.375649 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:21.375173    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsj8w/must-gather-xrn4l" event={"ID":"19d3903e-9d74-4df2-af63-6a897168f8c5","Type":"ContainerStarted","Data":"6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9"}
Jul 31 15:42:21.393599 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:21.393550    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nsj8w/must-gather-xrn4l" podStartSLOduration=2.254818992 podStartE2EDuration="8.393536272s" totalImagesPullingTime="6.13871728s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-31 15:42:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:42:21.392371446 +0000 UTC m=+3237.911795485" watchObservedRunningTime="2026-07-31 15:42:21.393536272 +0000 UTC m=+3237.912960309"
Jul 31 15:42:48.496474 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:48.496435    2561 generic.go:356] "Generic (PLEG): container finished" podID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerID="6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9" exitCode=0
Jul 31 15:42:48.496890 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:48.496509    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nsj8w/must-gather-xrn4l" event={"ID":"19d3903e-9d74-4df2-af63-6a897168f8c5","Type":"ContainerDied","Data":"6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9"}
Jul 31 15:42:48.496890 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:48.496803    2561 scope.go:117] "RemoveContainer" containerID="6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9"
Jul 31 15:42:49.333203 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:49.333172    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nsj8w_must-gather-xrn4l_19d3903e-9d74-4df2-af63-6a897168f8c5/gather/0.log"
Jul 31 15:42:52.731772 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:52.731743    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gfczg_7236d2d7-d084-40e7-9b8d-cd72aa4243b7/global-pull-secret-syncer/0.log"
Jul 31 15:42:52.878242 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:52.878216    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9j6k6_5719ef18-02ca-43e8-ac4a-342d57f1f6cc/konnectivity-agent/0.log"
Jul 31 15:42:52.948323 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:52.948294    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-192.ec2.internal_645b936c36ac9f10f261c827f21930d7/haproxy/0.log"
Jul 31 15:42:54.889579 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:54.889544    2561 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-must-gather-nsj8w/must-gather-xrn4l"]
Jul 31 15:42:54.890013 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:54.889748    2561 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-must-gather-nsj8w/must-gather-xrn4l" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerName="copy" containerID="cri-o://c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38" gracePeriod=2
Jul 31 15:42:54.894077 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:54.893834    2561 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-must-gather-nsj8w/must-gather-xrn4l"]
Jul 31 15:42:55.125371 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.125344    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nsj8w_must-gather-xrn4l_19d3903e-9d74-4df2-af63-6a897168f8c5/copy/0.log"
Jul 31 15:42:55.127373 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.127357    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:55.129420 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:42:55.129398    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xrn4l\" is forbidden: User \"system:node:ip-10-0-129-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nsj8w\": no relationship found between node 'ip-10-0-129-192.ec2.internal' and this object" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:55.221973 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.221885    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d3903e-9d74-4df2-af63-6a897168f8c5-must-gather-output\") pod \"19d3903e-9d74-4df2-af63-6a897168f8c5\" (UID: \"19d3903e-9d74-4df2-af63-6a897168f8c5\") "
Jul 31 15:42:55.221973 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.221928    2561 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-st7nk\" (UniqueName: \"kubernetes.io/projected/19d3903e-9d74-4df2-af63-6a897168f8c5-kube-api-access-st7nk\") pod \"19d3903e-9d74-4df2-af63-6a897168f8c5\" (UID: \"19d3903e-9d74-4df2-af63-6a897168f8c5\") "
Jul 31 15:42:55.223178 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.223137    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19d3903e-9d74-4df2-af63-6a897168f8c5-must-gather-output" (OuterVolumeSpecName: "must-gather-output") pod "19d3903e-9d74-4df2-af63-6a897168f8c5" (UID: "19d3903e-9d74-4df2-af63-6a897168f8c5"). InnerVolumeSpecName "must-gather-output". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 31 15:42:55.224181 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.224141    2561 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19d3903e-9d74-4df2-af63-6a897168f8c5-kube-api-access-st7nk" (OuterVolumeSpecName: "kube-api-access-st7nk") pod "19d3903e-9d74-4df2-af63-6a897168f8c5" (UID: "19d3903e-9d74-4df2-af63-6a897168f8c5"). InnerVolumeSpecName "kube-api-access-st7nk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 31 15:42:55.322982 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.322940    2561 reconciler_common.go:299] "Volume detached for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/19d3903e-9d74-4df2-af63-6a897168f8c5-must-gather-output\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:42:55.322982 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.322974    2561 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-st7nk\" (UniqueName: \"kubernetes.io/projected/19d3903e-9d74-4df2-af63-6a897168f8c5-kube-api-access-st7nk\") on node \"ip-10-0-129-192.ec2.internal\" DevicePath \"\""
Jul 31 15:42:55.526939 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.526911    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-must-gather-nsj8w_must-gather-xrn4l_19d3903e-9d74-4df2-af63-6a897168f8c5/copy/0.log"
Jul 31 15:42:55.527304 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.527280    2561 generic.go:356] "Generic (PLEG): container finished" podID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerID="c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38" exitCode=143
Jul 31 15:42:55.527423 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.527326    2561 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:55.527423 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.527337    2561 scope.go:117] "RemoveContainer" containerID="c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38"
Jul 31 15:42:55.529958 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:42:55.529930    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xrn4l\" is forbidden: User \"system:node:ip-10-0-129-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nsj8w\": no relationship found between node 'ip-10-0-129-192.ec2.internal' and this object" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:55.535391 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.535371    2561 scope.go:117] "RemoveContainer" containerID="6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9"
Jul 31 15:42:55.537525 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:42:55.537497    2561 status_manager.go:1018] "Failed to get status for pod" err="pods \"must-gather-xrn4l\" is forbidden: User \"system:node:ip-10-0-129-192.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-must-gather-nsj8w\": no relationship found between node 'ip-10-0-129-192.ec2.internal' and this object" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" pod="openshift-must-gather-nsj8w/must-gather-xrn4l"
Jul 31 15:42:55.547533 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.547514    2561 scope.go:117] "RemoveContainer" containerID="c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38"
Jul 31 15:42:55.547839 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:42:55.547815    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38\": container with ID starting with c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38 not found: ID does not exist" containerID="c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38"
Jul 31 15:42:55.547939 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.547845    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38"} err="failed to get container status \"c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38\": rpc error: code = NotFound desc = could not find container \"c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38\": container with ID starting with c9c4536429b26ec950584b352856cd9ba595744f6ddf952a55651ddab875af38 not found: ID does not exist"
Jul 31 15:42:55.547939 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.547865    2561 scope.go:117] "RemoveContainer" containerID="6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9"
Jul 31 15:42:55.548112 ip-10-0-129-192 kubenswrapper[2561]: E0731 15:42:55.548094    2561 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9\": container with ID starting with 6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9 not found: ID does not exist" containerID="6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9"
Jul 31 15:42:55.548180 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.548121    2561 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9"} err="failed to get container status \"6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9\": rpc error: code = NotFound desc = could not find container \"6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9\": container with ID starting with 6f90d2c9d7753e391056784111172188810c8a86e32d08eb5fe7c7be308f21e9 not found: ID does not exist"
Jul 31 15:42:55.960706 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:55.960625    2561 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" path="/var/lib/kubelet/pods/19d3903e-9d74-4df2-af63-6a897168f8c5/volumes"
Jul 31 15:42:57.622243 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:57.622197    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9mdg5_da6e8069-aaf2-46b5-8e2c-502580f70286/kube-state-metrics/0.log"
Jul 31 15:42:57.650505 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:57.650483    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9mdg5_da6e8069-aaf2-46b5-8e2c-502580f70286/kube-rbac-proxy-main/0.log"
Jul 31 15:42:57.678970 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:57.678932    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-6bd9664c4-9mdg5_da6e8069-aaf2-46b5-8e2c-502580f70286/kube-rbac-proxy-self/0.log"
Jul 31 15:42:57.710524 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:57.710494    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-57bf66bc-klhxq_252aefbd-e10e-4d56-86be-f9fdc401ae16/metrics-server/0.log"
Jul 31 15:42:57.741790 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:57.741759    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-6dffbbd67-t6zk8_51073248-1761-40d8-82b9-d54cbac2e7e7/monitoring-plugin/0.log"
Jul 31 15:42:57.878224 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:57.878097    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2bqd_f8a1604f-8087-41ab-b091-44868d7ad176/node-exporter/0.log"
Jul 31 15:42:57.900464 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:57.900435    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2bqd_f8a1604f-8087-41ab-b091-44868d7ad176/kube-rbac-proxy/0.log"
Jul 31 15:42:57.924576 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:57.924547    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d2bqd_f8a1604f-8087-41ab-b091-44868d7ad176/init-textfile/0.log"
Jul 31 15:42:58.141648 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.141571    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d30abdf-c7b4-4dbf-a2f4-e632fb16914d/prometheus/0.log"
Jul 31 15:42:58.166089 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.166060    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d30abdf-c7b4-4dbf-a2f4-e632fb16914d/config-reloader/0.log"
Jul 31 15:42:58.194007 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.193984    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d30abdf-c7b4-4dbf-a2f4-e632fb16914d/thanos-sidecar/0.log"
Jul 31 15:42:58.218063 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.218036    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d30abdf-c7b4-4dbf-a2f4-e632fb16914d/kube-rbac-proxy-web/0.log"
Jul 31 15:42:58.247802 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.247774    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d30abdf-c7b4-4dbf-a2f4-e632fb16914d/kube-rbac-proxy/0.log"
Jul 31 15:42:58.272191 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.272148    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d30abdf-c7b4-4dbf-a2f4-e632fb16914d/kube-rbac-proxy-thanos/0.log"
Jul 31 15:42:58.298360 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.298316    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_9d30abdf-c7b4-4dbf-a2f4-e632fb16914d/init-config-reloader/0.log"
Jul 31 15:42:58.680411 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.680385    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fc555f85b-lnp49_0db1694b-5738-4896-baa6-7385e719200f/thanos-query/0.log"
Jul 31 15:42:58.712980 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.712954    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fc555f85b-lnp49_0db1694b-5738-4896-baa6-7385e719200f/kube-rbac-proxy-web/0.log"
Jul 31 15:42:58.742374 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.742346    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fc555f85b-lnp49_0db1694b-5738-4896-baa6-7385e719200f/kube-rbac-proxy/0.log"
Jul 31 15:42:58.780820 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.780789    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fc555f85b-lnp49_0db1694b-5738-4896-baa6-7385e719200f/prom-label-proxy/0.log"
Jul 31 15:42:58.805953 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.805929    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fc555f85b-lnp49_0db1694b-5738-4896-baa6-7385e719200f/kube-rbac-proxy-rules/0.log"
Jul 31 15:42:58.829760 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:42:58.829736    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-5fc555f85b-lnp49_0db1694b-5738-4896-baa6-7385e719200f/kube-rbac-proxy-metrics/0.log"
Jul 31 15:43:00.014556 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:00.014513    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-f77f48c6d-ts756_f353439d-f0e7-4580-86a9-6719172fbcde/networking-console-plugin/0.log"
Jul 31 15:43:00.467515 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:00.467421    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/1.log"
Jul 31 15:43:00.471831 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:00.471805    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-5ccd6b445f-d892t_865fb521-3311-48c8-9f85-3b722f1ad540/console-operator/2.log"
Jul 31 15:43:00.914565 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:00.914531    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-77c974ffcf-hvbbv_3680a77c-3f4d-4032-987f-7cb4ab413ccc/download-server/0.log"
Jul 31 15:43:01.066906 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.066873    2561 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"]
Jul 31 15:43:01.067309 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.067202    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerName="copy"
Jul 31 15:43:01.067309 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.067214    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerName="copy"
Jul 31 15:43:01.067309 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.067231    2561 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerName="gather"
Jul 31 15:43:01.067309 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.067236    2561 state_mem.go:107] "Deleted CPUSet assignment" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerName="gather"
Jul 31 15:43:01.067309 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.067302    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerName="gather"
Jul 31 15:43:01.067472 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.067312    2561 memory_manager.go:368] "RemoveStaleState removing state" podUID="19d3903e-9d74-4df2-af63-6a897168f8c5" containerName="copy"
Jul 31 15:43:01.071099 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.071080    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.073537 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.073511    2561 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-cxx9f\"/\"default-dockercfg-rzbd9\""
Jul 31 15:43:01.074148 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.074130    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cxx9f\"/\"openshift-service-ca.crt\""
Jul 31 15:43:01.074265 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.074131    2561 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-cxx9f\"/\"kube-root-ca.crt\""
Jul 31 15:43:01.087069 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.087044    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"]
Jul 31 15:43:01.170309 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.170208    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-sys\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.170309 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.170257    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-proc\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.170536 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.170378    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xqwd\" (UniqueName: \"kubernetes.io/projected/ced5a483-531d-4a10-b280-691a9019295f-kube-api-access-7xqwd\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.170536 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.170431    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-lib-modules\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.170536 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.170455    2561 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-podres\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.271885 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.271843    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xqwd\" (UniqueName: \"kubernetes.io/projected/ced5a483-531d-4a10-b280-691a9019295f-kube-api-access-7xqwd\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.272095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.271895    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-lib-modules\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.272095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.271917    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-podres\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.272095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.271947    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-sys\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.272095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.271974    2561 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-proc\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.272095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.272049    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-proc\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.272095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.272056    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-lib-modules\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.272095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.272072    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-sys\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.272095 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.272073    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/ced5a483-531d-4a10-b280-691a9019295f-podres\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.282490 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.282467    2561 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xqwd\" (UniqueName: \"kubernetes.io/projected/ced5a483-531d-4a10-b280-691a9019295f-kube-api-access-7xqwd\") pod \"perf-node-gather-daemonset-4nkdt\" (UID: \"ced5a483-531d-4a10-b280-691a9019295f\") " pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.380862 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.380821    2561 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:01.512929 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.512904    2561 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"]
Jul 31 15:43:01.515975 ip-10-0-129-192 kubenswrapper[2561]: W0731 15:43:01.515947    2561 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podced5a483_531d_4a10_b280_691a9019295f.slice/crio-dae10868e5216adabcd890587f9b7e8c5937c16462c22db41088ac662b30357d WatchSource:0}: Error finding container dae10868e5216adabcd890587f9b7e8c5937c16462c22db41088ac662b30357d: Status 404 returned error can't find the container with id dae10868e5216adabcd890587f9b7e8c5937c16462c22db41088ac662b30357d
Jul 31 15:43:01.553908 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:01.553880    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt" event={"ID":"ced5a483-531d-4a10-b280-691a9019295f","Type":"ContainerStarted","Data":"dae10868e5216adabcd890587f9b7e8c5937c16462c22db41088ac662b30357d"}
Jul 31 15:43:02.483824 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:02.483789    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hcnkz_51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5/dns/0.log"
Jul 31 15:43:02.506712 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:02.506683    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-hcnkz_51ef06e7-ca6d-44e7-ab06-0fa2d0adfae5/kube-rbac-proxy/0.log"
Jul 31 15:43:02.559493 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:02.559460    2561 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt" event={"ID":"ced5a483-531d-4a10-b280-691a9019295f","Type":"ContainerStarted","Data":"d40283c8b5603ca75401bb616bd0d8690bbe5459ecc287d072dd8ba3bb22b396"}
Jul 31 15:43:02.559671 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:02.559529    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:02.579624 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:02.579574    2561 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt" podStartSLOduration=1.579557651 podStartE2EDuration="1.579557651s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-31 15:43:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-31 15:43:02.57878056 +0000 UTC m=+3279.098204597" watchObservedRunningTime="2026-07-31 15:43:02.579557651 +0000 UTC m=+3279.098981689"
Jul 31 15:43:02.635775 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:02.635749    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-wr8xz_98896ea7-ed6d-451c-a61b-e74ab2b1cdf9/dns-node-resolver/0.log"
Jul 31 15:43:03.082833 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:03.082801    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-d85694948-dfr68_6d566451-0092-4630-9010-94495d450748/registry/0.log"
Jul 31 15:43:03.133805 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:03.133777    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lz89t_68e2f225-4cc2-465b-b01b-6f1e8142b94b/node-ca/0.log"
Jul 31 15:43:04.248008 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:04.247983    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-mwkpt_27804094-6280-4f24-a5c7-0a0dc5ff6561/serve-healthcheck-canary/0.log"
Jul 31 15:43:04.637184 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:04.637088    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-cx286_bb00f3ae-bc95-452c-b977-0a857611efcd/insights-operator/1.log"
Jul 31 15:43:04.637348 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:04.637187    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-77696685b-cx286_bb00f3ae-bc95-452c-b977-0a857611efcd/insights-operator/0.log"
Jul 31 15:43:04.658204 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:04.658148    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4b8hs_ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a/kube-rbac-proxy/0.log"
Jul 31 15:43:04.680784 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:04.680757    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4b8hs_ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a/exporter/0.log"
Jul 31 15:43:04.709147 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:04.709121    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4b8hs_ad36a1aa-e3c4-4af6-ad0f-038b2e66ae0a/extractor/0.log"
Jul 31 15:43:07.727879 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:07.727837    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-szhr4_6ded908c-471d-4137-b304-ecc4d44d8c69/s3-init/0.log"
Jul 31 15:43:07.744512 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:07.744488    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-init-szhr4_6ded908c-471d-4137-b304-ecc4d44d8c69/download-hf-model/0.log"
Jul 31 15:43:07.772794 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:07.772762    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-b67rd_a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1/s3-tls-init-custom/0.log"
Jul 31 15:43:07.789463 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:07.789437    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-custom-b67rd_a51d69c4-6bfa-4b5b-9fd6-ebb8c057a0d1/download-hf-model/0.log"
Jul 31 15:43:07.849130 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:07.849097    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-qrnlt_4117c74d-5256-458c-bf65-e70901742bed/s3-tls-init-serving/0.log"
Jul 31 15:43:07.867691 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:07.867664    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_s3-tls-init-serving-qrnlt_4117c74d-5256-458c-bf65-e70901742bed/download-hf-model/0.log"
Jul 31 15:43:08.573626 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:08.573598    2561 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-cxx9f/perf-node-gather-daemonset-4nkdt"
Jul 31 15:43:12.160065 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:12.160035    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-mrt5c_d97d6c88-4d9b-4ca8-bc7c-87e814b576cf/migrator/0.log"
Jul 31 15:43:12.181997 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:12.181971    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-68bc745d87-mrt5c_d97d6c88-4d9b-4ca8-bc7c-87e814b576cf/graceful-termination/0.log"
Jul 31 15:43:13.588345 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:13.588316    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ctdq_bf493cbb-ad21-4177-b194-d73d297a4cb0/kube-multus-additional-cni-plugins/0.log"
Jul 31 15:43:13.616227 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:13.616199    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ctdq_bf493cbb-ad21-4177-b194-d73d297a4cb0/egress-router-binary-copy/0.log"
Jul 31 15:43:13.638797 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:13.638773    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ctdq_bf493cbb-ad21-4177-b194-d73d297a4cb0/cni-plugins/0.log"
Jul 31 15:43:13.662766 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:13.662742    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ctdq_bf493cbb-ad21-4177-b194-d73d297a4cb0/bond-cni-plugin/0.log"
Jul 31 15:43:13.687654 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:13.687632    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ctdq_bf493cbb-ad21-4177-b194-d73d297a4cb0/routeoverride-cni/0.log"
Jul 31 15:43:13.709620 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:13.709589    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ctdq_bf493cbb-ad21-4177-b194-d73d297a4cb0/whereabouts-cni-bincopy/0.log"
Jul 31 15:43:13.732992 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:13.732965    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-7ctdq_bf493cbb-ad21-4177-b194-d73d297a4cb0/whereabouts-cni/0.log"
Jul 31 15:43:13.970309 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:13.970280    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-cx6kt_2203e737-ecf1-46d7-81ea-be5ee9c0d476/kube-multus/0.log"
Jul 31 15:43:14.038795 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.038765    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qpj79_bcd6660e-bbc4-47a2-94aa-d65b15d96247/network-metrics-daemon/0.log"
Jul 31 15:43:14.060754 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.060729    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qpj79_bcd6660e-bbc4-47a2-94aa-d65b15d96247/kube-rbac-proxy/0.log"
Jul 31 15:43:14.841626 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.841597    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2dtc_598105ef-f9a1-43c1-abf2-827929b9291b/ovn-controller/0.log"
Jul 31 15:43:14.864597 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.864566    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2dtc_598105ef-f9a1-43c1-abf2-827929b9291b/ovn-acl-logging/0.log"
Jul 31 15:43:14.893287 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.893261    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2dtc_598105ef-f9a1-43c1-abf2-827929b9291b/kube-rbac-proxy-node/0.log"
Jul 31 15:43:14.917208 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.917180    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2dtc_598105ef-f9a1-43c1-abf2-827929b9291b/kube-rbac-proxy-ovn-metrics/0.log"
Jul 31 15:43:14.941629 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.941599    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2dtc_598105ef-f9a1-43c1-abf2-827929b9291b/northd/0.log"
Jul 31 15:43:14.965207 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.965149    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2dtc_598105ef-f9a1-43c1-abf2-827929b9291b/nbdb/0.log"
Jul 31 15:43:14.989488 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:14.989453    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2dtc_598105ef-f9a1-43c1-abf2-827929b9291b/sbdb/0.log"
Jul 31 15:43:15.035991 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:15.035958    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k2dtc_598105ef-f9a1-43c1-abf2-827929b9291b/ovnkube-controller/0.log"
Jul 31 15:43:16.505502 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:16.505473    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-source-65bdbbff45-88r5l_f3a91903-3e85-450e-a8af-670aaddd059b/check-endpoints/0.log"
Jul 31 15:43:16.562249 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:16.562219    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-ch9qv_73a4030c-d937-4fde-b65c-e026541f6cbe/network-check-target-container/0.log"
Jul 31 15:43:17.551249 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:17.551219    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-operator_iptables-alerter-9gzdg_4bd381bb-8a21-4c36-a8ce-bbf3ecebab43/iptables-alerter/0.log"
Jul 31 15:43:18.284377 ip-10-0-129-192 kubenswrapper[2561]: I0731 15:43:18.284350    2561 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-node-tuning-operator_tuned-dgg8r_35d8467c-ac8e-4e58-87c2-bf13e6faa093/tuned/0.log"