Jul 28 10:06:05.279190 ip-10-0-131-51 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 28 10:06:05.279204 ip-10-0-131-51 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 28 10:06:05.279213 ip-10-0-131-51 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 28 10:06:05.279555 ip-10-0-131-51 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 28 10:06:15.440760 ip-10-0-131-51 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 28 10:06:15.440778 ip-10-0-131-51 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot a25a74e7afad41568b995327d276b3a7 --
Jul 28 10:08:12.307262 ip-10-0-131-51 systemd[1]: Starting Kubernetes Kubelet...
Jul 28 10:08:12.732530 ip-10-0-131-51 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:08:12.732530 ip-10-0-131-51 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 28 10:08:12.732530 ip-10-0-131-51 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:08:12.732530 ip-10-0-131-51 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 28 10:08:12.732530 ip-10-0-131-51 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 28 10:08:12.733303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.732604    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 28 10:08:12.738102 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738079    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:08:12.738102 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738098    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:08:12.738102 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738103    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:08:12.738102 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738108    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738115    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738120    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738125    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738129    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738134    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738138    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738142    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738146    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738151    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738158    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738163    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738168    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738172    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738177    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738182    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738186    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738190    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738194    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738200    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:08:12.738372 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738206    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738210    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738214    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738219    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738223    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738228    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738231    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738236    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738240    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738245    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738249    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738254    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738258    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738263    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738269    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738273    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738278    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738282    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738287    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738291    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:08:12.739272 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738296    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738301    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738306    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738311    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738315    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738320    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738324    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738329    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738333    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738338    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738342    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738347    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738351    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738356    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738360    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738364    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738369    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738373    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738378    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738382    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:08:12.740212 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738386    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738391    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738395    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738402    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738408    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738413    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738419    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738424    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738430    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738436    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738443    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738448    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738454    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738459    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738464    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738469    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738473    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738478    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738482    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:08:12.741195 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738487    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738493    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738497    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738502    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738506    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738511    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738515    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738519    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738523    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738528    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738533    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738538    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738543    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738547    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738552    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738557    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.738561    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739847    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739859    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:08:12.742111 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739867    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739872    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739877    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739881    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739886    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739890    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739895    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739901    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739905    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739910    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739915    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739919    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739924    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739928    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739933    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739938    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739943    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739947    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739952    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:08:12.743011 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739957    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739961    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739969    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739976    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739981    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739988    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.739994    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740000    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740005    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740011    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740016    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740021    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740026    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740034    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740040    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740045    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740050    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740055    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740060    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:08:12.743639 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740065    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740069    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740075    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740080    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740085    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740090    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740094    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740099    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740104    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740108    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740113    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740117    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740122    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740127    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740132    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740136    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740141    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740145    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740151    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740155    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:08:12.744273 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740160    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740165    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740170    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740175    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740180    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740184    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740191    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740196    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740200    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740205    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740209    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740214    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740218    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740222    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740226    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740232    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740236    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740241    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740245    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740251    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:08:12.744923 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740256    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740261    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740266    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740270    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740275    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740280    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740284    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740289    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740294    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740299    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740304    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740308    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740313    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740318    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740323    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740327    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740331    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740336    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.740342    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741004    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:08:12.745588 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741014    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741019    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741024    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741029    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741035    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741040    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741044    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741050    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741055    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741060    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741065    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741070    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741074    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741079    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741084    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741088    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741093    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741098    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741102    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741107    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:08:12.746159 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741112    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741117    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741122    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741126    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741131    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741135    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741140    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741144    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741149    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741156    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741164    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741169    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741173    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741178    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741184    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741188    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741193    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741197    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741201    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:08:12.746810 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741205    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741210    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741215    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741219    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741223    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741228    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741233    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741237    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741242    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741247    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741251    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741256    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741261    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741266    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741274    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741281    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741286    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741291    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741295    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:08:12.747335 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741300    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741304    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741309    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741314    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741318    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741323    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741328    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741332    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741337    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741341    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741345    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741350    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741355    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741364    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741370    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741376    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741382    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741387    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741392    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741397    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:08:12.747908 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741401    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741405    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741410    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741415    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741419    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741424    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741429    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741433    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741438    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741443    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741447    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741451    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741456    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741461    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741465    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741470    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741475    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741480    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741484    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.741489    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:08:12.748429 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741598    2560 flags.go:64] FLAG: --address="0.0.0.0"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741609    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741618    2560 flags.go:64] FLAG: --anonymous-auth="true"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741626    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741634    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741640    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741647    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741654    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741660    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741666    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741672    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741678    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741684    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741689    2560 flags.go:64] FLAG: --cgroup-root=""
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741694    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741699    2560 flags.go:64] FLAG: --client-ca-file=""
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741705    2560 flags.go:64] FLAG: --cloud-provider="external"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741710    2560 flags.go:64] FLAG: --cluster-dns="[]"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741718    2560 flags.go:64] FLAG: --cluster-domain=""
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741723    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741729    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741735    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741740    2560 flags.go:64] FLAG: --container-log-max-files="5"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741747    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 28 10:08:12.748971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741752    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741757    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741763    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741768    2560 flags.go:64] FLAG: --contention-profiling="false"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741774    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741780    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741806    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741812    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741819    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741825    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741829    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741837    2560 flags.go:64] FLAG: --enable-load-reader="false"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741842    2560 flags.go:64] FLAG: --enable-server="true"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741847    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741855    2560 flags.go:64] FLAG: --event-burst="100"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741860    2560 flags.go:64] FLAG: --event-qps="50"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741865    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741872    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741877    2560 flags.go:64] FLAG: --eviction-hard=""
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741884    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741889    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741894    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741900    2560 flags.go:64] FLAG: --eviction-soft=""
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741905    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741911    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 28 10:08:12.749627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741916    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741921    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741925    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741931    2560 flags.go:64] FLAG: --fail-swap-on="true"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741936    2560 flags.go:64] FLAG: --feature-gates=""
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741943    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741948    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741953    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741958    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741964    2560 flags.go:64] FLAG: --healthz-port="10248"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741969    2560 flags.go:64] FLAG: --help="false"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741974    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-131-51.ec2.internal"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741980    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741986    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741991    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.741997    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742003    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742008    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742015    2560 flags.go:64] FLAG: --image-service-endpoint=""
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742020    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742025    2560 flags.go:64] FLAG: --kube-api-burst="100"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742030    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742036    2560 flags.go:64] FLAG: --kube-api-qps="50"
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742041    2560 flags.go:64] FLAG: --kube-reserved=""
Jul 28 10:08:12.750409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742046    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742061    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742066    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742071    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742076    2560 flags.go:64] FLAG: --lock-file=""
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742082    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742087    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742092    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742101    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742106    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742111    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742115    2560 flags.go:64] FLAG: --logging-format="text"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742120    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742125    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742130    2560 flags.go:64] FLAG: --manifest-url=""
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742134    2560 flags.go:64] FLAG: --manifest-url-header=""
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742141    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742147    2560 flags.go:64] FLAG: --max-open-files="1000000"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742154    2560 flags.go:64] FLAG: --max-pods="110"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742161    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742166    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742171    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742176    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742181    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742186    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 28 10:08:12.751077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742192    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742205    2560 flags.go:64] FLAG: --node-status-max-images="50"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742210    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742215    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742220    2560 flags.go:64] FLAG: --pod-cidr=""
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742225    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742235    2560 flags.go:64] FLAG: --pod-manifest-path=""
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742240    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742246    2560 flags.go:64] FLAG: --pods-per-core="0"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742259    2560 flags.go:64] FLAG: --port="10250"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742265    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742270    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0eef8752384c73948"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742275    2560 flags.go:64] FLAG: --qos-reserved=""
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742281    2560 flags.go:64] FLAG: --read-only-port="10255"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742286    2560 flags.go:64] FLAG: --register-node="true"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742291    2560 flags.go:64] FLAG: --register-with-taints=""
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742298    2560 flags.go:64] FLAG: --registry-burst="10"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742303    2560 flags.go:64] FLAG: --registry-qps="5"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742308    2560 flags.go:64] FLAG: --reserved-cpus=""
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742313    2560 flags.go:64] FLAG: --reserved-memory=""
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742320    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742326    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742331    2560 flags.go:64] FLAG: --rotate-certificates="false"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742336    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742341    2560 flags.go:64] FLAG: --runonce="false"
Jul 28 10:08:12.751766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742346    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742352    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742359    2560 flags.go:64] FLAG: --seccomp-default="false"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742364    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742369    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742375    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742380    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742386    2560 flags.go:64] FLAG: --storage-driver-password="root"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742396    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742402    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742406    2560 flags.go:64] FLAG: --storage-driver-user="root"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742412    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742417    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742422    2560 flags.go:64] FLAG: --system-cgroups=""
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742427    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742437    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742442    2560 flags.go:64] FLAG: --tls-cert-file=""
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742455    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742467    2560 flags.go:64] FLAG: --tls-min-version=""
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742472    2560 flags.go:64] FLAG: --tls-private-key-file=""
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742477    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742482    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742487    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742492    2560 flags.go:64] FLAG: --v="2"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742499    2560 flags.go:64] FLAG: --version="false"
Jul 28 10:08:12.752451 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742506    2560 flags.go:64] FLAG: --vmodule=""
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742512    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.742518    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742709    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742716    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742721    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742727    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742732    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742736    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742743    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742747    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742752    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742756    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742761    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742765    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742772    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742777    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742782    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742804    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742809    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:08:12.753173 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742814    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742818    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742823    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742827    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742834    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742846    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742851    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742858    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742863    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742868    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742873    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742878    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742883    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742888    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742893    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742898    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742902    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742907    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742911    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:08:12.753697 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742916    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742920    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742927    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742931    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742936    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742941    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742945    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742950    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742956    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742961    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742965    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742970    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742974    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742979    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742984    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742988    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742993    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.742997    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743002    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743007    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:08:12.754220 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743012    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743016    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743021    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743025    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743029    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743034    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743038    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743043    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743047    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743052    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743057    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743061    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743066    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743073    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743081    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743085    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743090    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743095    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743100    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743107    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:08:12.754740 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743114    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743120    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743125    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743131    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743136    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743141    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743146    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743151    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743156    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743161    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743165    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743170    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743175    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743180    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743184    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743189    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743193    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743198    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743202    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743207    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:08:12.755292 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743211    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743215    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.743220    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.743996    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.751178    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.751192    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.752112    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.752121    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752179    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752184    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752188    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752191    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752194    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752197    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752200    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:08:12.755918 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752203    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752206    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752209    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752213    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752218    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752221    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752224    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752227    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752230    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752233    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752237    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752240    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752242    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752245    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752248    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752252    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752255    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752258    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752260    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:08:12.756322 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752263    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752266    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752269    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752272    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752277    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752281    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752285    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752288    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752292    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752295    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752298    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752301    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752304    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752307    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752310    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752313    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752316    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752319    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752322    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:08:12.756847 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752325    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752329    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752332    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752335    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752338    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752341    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752344    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752347    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752350    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752353    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752355    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752358    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752361    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752364    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752367    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752370    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752374    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752377    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752380    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752383    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:08:12.757348 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752386    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752388    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752391    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752394    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752396    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752399    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752402    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752404    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752407    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752410    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752412    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752415    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752418    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752421    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752423    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752426    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752430    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752434    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752437    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:08:12.757905 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752440    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752443    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752445    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752448    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752451    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752454    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752457    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752460    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752462    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752466    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752469    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752472    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752475    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752478    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752481    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:08:12.758393 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.752486    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752602    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752607    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752611    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752614    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752617    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752620    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752624    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752627    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752630    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752632    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752635    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752638    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752641    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752644    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752647    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752651    2560 feature_gate.go:328] unrecognized feature gate: Example
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752654    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752658    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752661    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 28 10:08:12.758805 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752664    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752667    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752670    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752673    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752676    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752678    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752681    2560 feature_gate.go:328] unrecognized feature gate: Example2
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752684    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752687    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752689    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752692    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752695    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752697    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752700    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752705    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752708    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752712    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752715    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752718    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752721    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 28 10:08:12.759320 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752723    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752726    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752729    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752732    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752735    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752737    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752740    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752744    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752747    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752751    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752754    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752757    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752760    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752775    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752779    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752782    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752785    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752789    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752794    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 28 10:08:12.759861 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752799    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752803    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752806    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752809    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752812    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752815    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752818    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752821    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752824    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752826    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752829    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752832    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752834    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752837    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752840    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752842    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752846    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752850    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752853    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752855    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 28 10:08:12.760349 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752858    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752861    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752863    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752867    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752870    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752873    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752876    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752879    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752881    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752884    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752887    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752890    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752893    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752896    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752899    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752901    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752904    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752907    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752910    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752912    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 28 10:08:12.760953 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:12.752915    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 28 10:08:12.761500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.752920    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 28 10:08:12.761500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.753024    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 28 10:08:12.761500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.756194    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 28 10:08:12.761500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.757656    2560 server.go:1019] "Starting client certificate rotation"
Jul 28 10:08:12.761500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.757765    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:08:12.761500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.757811    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 28 10:08:12.763278 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.763251    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:08:12.764488 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.764467    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 28 10:08:12.771523 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.771505    2560 log.go:25] "Validated CRI v1 runtime API"
Jul 28 10:08:12.773632 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.773618    2560 log.go:25] "Validated CRI v1 image API"
Jul 28 10:08:12.774491 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.774475    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 28 10:08:12.777201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.777178    2560 fs.go:135] Filesystem UUIDs: map[407b7aea-ff31-4520-a8fd-eb158e126ffd:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 915e38c8-8c7c-4f7a-9336-6cb94eb36986:/dev/nvme0n1p3]
Jul 28 10:08:12.777274 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.777199    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 28 10:08:12.782590 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.782485    2560 manager.go:216] Machine: {Timestamp:2026-07-28 10:08:12.781153847 +0000 UTC m=+0.356346770 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3086750 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2963d9e45c1f57ca7193d28e1a807f SystemUUID:ec2963d9-e45c-1f57-ca71-93d28e1a807f BootID:a25a74e7-afad-4156-8b99-5327d276b3a7 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f2:c5:f2:9b:89 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f2:c5:f2:9b:89 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:f0:93:a2:8b:17 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 28 10:08:12.782590 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.782583    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 28 10:08:12.782709 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.782655    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 28 10:08:12.786061 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.786033    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 28 10:08:12.786200 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.786064    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-131-51.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 28 10:08:12.786836 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.786825    2560 topology_manager.go:138] "Creating topology manager with none policy"
Jul 28 10:08:12.786869 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.786839    2560 container_manager_linux.go:309] "Creating device plugin manager"
Jul 28 10:08:12.786869 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.786847    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:08:12.787545 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.787532    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 28 10:08:12.788417 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.788403    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 28 10:08:12.790369 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.790355    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:08:12.790513 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.790502    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 28 10:08:12.790551 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.790541    2560 kubelet.go:486] "Attempting to sync node with API server"
Jul 28 10:08:12.790590 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.790570    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 28 10:08:12.790627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.790589    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 28 10:08:12.790627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.790603    2560 kubelet.go:398] "Adding apiserver pod source"
Jul 28 10:08:12.790706 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.790695    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 28 10:08:12.791311 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.791297    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:08:12.791361 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.791319    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 28 10:08:12.791706 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.791677    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:08:12.796513 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.796494    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 28 10:08:12.798418 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.798404    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 28 10:08:12.798477 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.798436    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 28 10:08:12.800066 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800050    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 28 10:08:12.800066 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800067    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800074    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800080    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800089    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800095    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800103    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800109    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800117    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800123    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800132    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 28 10:08:12.800211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.800138    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 28 10:08:12.801044 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.801033    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 28 10:08:12.801044 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.801043    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 28 10:08:12.803265 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.803243    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-131-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 28 10:08:12.803343 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.803272    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-131-51.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 28 10:08:12.803343 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.803296    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 28 10:08:12.804240 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.804227    2560 server.go:1268] "Started kubelet"
Jul 28 10:08:12.804331 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.804306    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 28 10:08:12.804447 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.804396    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 28 10:08:12.804509 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.804472    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 28 10:08:12.805142 ip-10-0-131-51 systemd[1]: Started Kubernetes Kubelet.
Jul 28 10:08:12.805862 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.805630    2560 server.go:310] "Adding debug handlers to kubelet server"
Jul 28 10:08:12.806204 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.806188    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 28 10:08:12.808597 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.808575    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 28 10:08:12.813321 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.813291    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 28 10:08:12.813321 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.813310    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 28 10:08:12.813479 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.813388    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:12.814084 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814057    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 28 10:08:12.814084 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814058    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 28 10:08:12.814229 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814110    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 28 10:08:12.814291 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814261    2560 reconstruct.go:97] "Volume reconstruction finished"
Jul 28 10:08:12.814291 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814274    2560 reconciler.go:29] "Reconciler: start to sync state"
Jul 28 10:08:12.814414 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814396    2560 factory.go:55] Registering systemd factory
Jul 28 10:08:12.814522 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814441    2560 factory.go:223] Registration of the systemd container factory successfully
Jul 28 10:08:12.814697 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814680    2560 factory.go:153] Registering CRI-O factory
Jul 28 10:08:12.814697 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814699    2560 factory.go:223] Registration of the crio container factory successfully
Jul 28 10:08:12.814860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814780    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 28 10:08:12.814860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814810    2560 factory.go:103] Registering Raw factory
Jul 28 10:08:12.814860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.814831    2560 manager.go:1196] Started watching for new ooms in manager
Jul 28 10:08:12.815177 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.815166    2560 manager.go:349] Starting recovery of all containers
Jul 28 10:08:12.815811 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.815782    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-131-51.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 28 10:08:12.816321 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.816288    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 28 10:08:12.817068 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.815774    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-51.ec2.internal.18c66c38a2cb94fb  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-51.ec2.internal,UID:ip-10-0-131-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-131-51.ec2.internal,},FirstTimestamp:2026-07-28 10:08:12.804199675 +0000 UTC m=+0.379392597,LastTimestamp:2026-07-28 10:08:12.804199675 +0000 UTC m=+0.379392597,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-51.ec2.internal,}"
Jul 28 10:08:12.822613 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.822588    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sh8q9"
Jul 28 10:08:12.828044 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.828024    2560 manager.go:354] Recovery completed
Jul 28 10:08:12.828993 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.828971    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sh8q9"
Jul 28 10:08:12.832737 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.832722    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:08:12.836472 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.836455    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:08:12.836555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.836483    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:08:12.836555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.836496    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:08:12.837016 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.837002    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 28 10:08:12.837016 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.837014    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 28 10:08:12.837116 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.837029    2560 state_mem.go:36] "Initialized new in-memory state store"
Jul 28 10:08:12.837116 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.837077    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:08:12.837864 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.837848    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:08:12.837922 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.837875    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:08:12.837922 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.837885    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:08:12.838379 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.838311    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-131-51.ec2.internal.18c66c38a4b7fd89  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-131-51.ec2.internal,UID:ip-10-0-131-51.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-131-51.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-131-51.ec2.internal,},FirstTimestamp:2026-07-28 10:08:12.836470153 +0000 UTC m=+0.411663078,LastTimestamp:2026-07-28 10:08:12.836470153 +0000 UTC m=+0.411663078,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-131-51.ec2.internal,}"
Jul 28 10:08:12.838761 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.838748    2560 policy_none.go:49] "None policy: Start"
Jul 28 10:08:12.838810 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.838766    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 28 10:08:12.838810 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.838778    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 28 10:08:12.840034 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.840023    2560 policy_none.go:47] "Start"
Jul 28 10:08:12.876480 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.876361    2560 manager.go:337] "Starting Device Plugin manager"
Jul 28 10:08:12.876480 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.876477    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 28 10:08:12.876639 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.876487    2560 server.go:87] "Starting device plugin registration server"
Jul 28 10:08:12.876746 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.876720    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 28 10:08:12.876746 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.876733    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 28 10:08:12.876870 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.876853    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 28 10:08:12.876947 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.876935    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 28 10:08:12.876996 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.876948    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 28 10:08:12.877501 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.877481    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 28 10:08:12.877589 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.877536    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:12.970746 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.970717    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 28 10:08:12.972156 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.971914    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 28 10:08:12.972156 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.971951    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 28 10:08:12.972156 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.971976    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 28 10:08:12.972156 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.972051    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 28 10:08:12.974415 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.974394    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:08:12.977391 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.977375    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:08:12.978163 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.978147    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:08:12.978248 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.978169    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:08:12.978248 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.978179    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:08:12.978248 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.978199    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-131-51.ec2.internal"
Jul 28 10:08:12.985340 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:12.985324    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-131-51.ec2.internal"
Jul 28 10:08:12.985432 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.985349    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-131-51.ec2.internal\": node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:12.997199 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:12.997181    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.072774 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.072746    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal"]
Jul 28 10:08:13.072857 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.072820    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:08:13.074410 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.074386    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:08:13.074506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.074423    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:08:13.074506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.074438    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:08:13.075698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.075685    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:08:13.075820 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.075802    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.075878 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.075835    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:08:13.076554 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.076537    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:08:13.076642 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.076560    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:08:13.076642 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.076569    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:08:13.076642 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.076584    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:08:13.076642 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.076596    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:08:13.076780 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.076584    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:08:13.078019 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.078003    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.078077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.078028    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 28 10:08:13.078777 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.078759    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientMemory"
Jul 28 10:08:13.078890 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.078790    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasNoDiskPressure"
Jul 28 10:08:13.078890 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.078806    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeHasSufficientPID"
Jul 28 10:08:13.097550 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.097527    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.102542 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.102526    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-51.ec2.internal\" not found" node="ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.107367 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.107349    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-131-51.ec2.internal\" not found" node="ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.116827 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.116795    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf42ad45c184732bda36fba61d2a3c5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-51.ec2.internal\" (UID: \"2cf42ad45c184732bda36fba61d2a3c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.116916 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.116841    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7eb04e870c582d25e244a866d995f954-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal\" (UID: \"7eb04e870c582d25e244a866d995f954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.116916 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.116860    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb04e870c582d25e244a866d995f954-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal\" (UID: \"7eb04e870c582d25e244a866d995f954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.198627 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.198606    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.217546 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.217527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf42ad45c184732bda36fba61d2a3c5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-51.ec2.internal\" (UID: \"2cf42ad45c184732bda36fba61d2a3c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.217602 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.217550    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7eb04e870c582d25e244a866d995f954-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal\" (UID: \"7eb04e870c582d25e244a866d995f954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.217602 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.217569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb04e870c582d25e244a866d995f954-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal\" (UID: \"7eb04e870c582d25e244a866d995f954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.217602 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.217597    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/7eb04e870c582d25e244a866d995f954-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal\" (UID: \"7eb04e870c582d25e244a866d995f954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.217715 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.217616    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/7eb04e870c582d25e244a866d995f954-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal\" (UID: \"7eb04e870c582d25e244a866d995f954\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.217715 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.217626    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/2cf42ad45c184732bda36fba61d2a3c5-config\") pod \"kube-apiserver-proxy-ip-10-0-131-51.ec2.internal\" (UID: \"2cf42ad45c184732bda36fba61d2a3c5\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.299731 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.299676    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.406507 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.406489    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.406566 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.406487    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.411636 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.410591    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:13.507620 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.507595    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.609491 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.609436    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.713744 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.713704    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.736211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.736192    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:08:13.756833 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.756801    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 28 10:08:13.756943 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.756925    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:08:13.756983 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.756964    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 28 10:08:13.814171 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.814144    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.833516 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.833482    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-27 10:03:13 +0000 UTC" deadline="2028-01-06 07:28:54.612706503 +0000 UTC"
Jul 28 10:08:13.833516 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.833515    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="12645h20m40.779196749s"
Jul 28 10:08:13.850085 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.850060    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:08:13.914952 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:13.914926    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-131-51.ec2.internal\" not found"
Jul 28 10:08:13.920400 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.920379    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:08:13.986136 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:13.986109    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cf42ad45c184732bda36fba61d2a3c5.slice/crio-adba13b24bd810cf0f3f554923c3efb89e403a50b9ba1de02680e1228d34817b WatchSource:0}: Error finding container adba13b24bd810cf0f3f554923c3efb89e403a50b9ba1de02680e1228d34817b: Status 404 returned error can't find the container with id adba13b24bd810cf0f3f554923c3efb89e403a50b9ba1de02680e1228d34817b
Jul 28 10:08:13.986622 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:13.986604    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7eb04e870c582d25e244a866d995f954.slice/crio-ebd811e3a50fd9b5d47182171fabf8f2f139a5c4cba8ae20fb6117977a83f77a WatchSource:0}: Error finding container ebd811e3a50fd9b5d47182171fabf8f2f139a5c4cba8ae20fb6117977a83f77a: Status 404 returned error can't find the container with id ebd811e3a50fd9b5d47182171fabf8f2f139a5c4cba8ae20fb6117977a83f77a
Jul 28 10:08:13.992794 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:13.992767    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:08:14.014017 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.013995    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:14.027145 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.027125    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:08:14.028359 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.028347    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal"
Jul 28 10:08:14.039987 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.039959    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 28 10:08:14.604276 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.604250    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:08:14.793664 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.793632    2560 apiserver.go:52] "Watching apiserver"
Jul 28 10:08:14.801819 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.800837    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 28 10:08:14.801819 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.801370    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal","openshift-multus/multus-additional-cni-plugins-2fljt","openshift-network-diagnostics/network-check-target-lm85v","kube-system/global-pull-secret-syncer-t6srt","openshift-image-registry/node-ca-6zfzk","openshift-multus/multus-6h4hc","openshift-multus/network-metrics-daemon-vxtwq","openshift-network-operator/iptables-alerter-q2w92","openshift-ovn-kubernetes/ovnkube-node-9cmbd","kube-system/konnectivity-agent-szf6k","kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg","openshift-cluster-node-tuning-operator/tuned-t2z2v","openshift-dns/node-resolver-n72pd"]
Jul 28 10:08:14.803828 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.803357    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:14.803828 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.803446    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:14.805619 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.805593    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.806841 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.806817    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:14.806942 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.806906    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:14.808181 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.808129    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.808285 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.808196    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:14.808353 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.808289    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.808455 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.808436    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.808510 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.808468    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hlc5n\""
Jul 28 10:08:14.809823 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.809534    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.812943 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.810573    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.812943 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.812207    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.812943 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.812775    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 28 10:08:14.812943 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.812785    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vs8gm\""
Jul 28 10:08:14.812943 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.812807    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.813253 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.813123    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.813333 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.813314    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.813444 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.813364    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 28 10:08:14.813494 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.813460    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.813760 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.813585    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 28 10:08:14.813760 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.813681    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-s9fw5\""
Jul 28 10:08:14.813760 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.813740    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.814297 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.814251    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 28 10:08:14.814534 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.814509    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.815124 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.814891    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-p57kj\""
Jul 28 10:08:14.815124 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.814949    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 28 10:08:14.815360 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.815332    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.816022 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.815878    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 28 10:08:14.817711 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.816671    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 28 10:08:14.817711 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.816958    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.817711 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.816987    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wl5rq\""
Jul 28 10:08:14.817711 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.817233    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 28 10:08:14.817711 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.817280    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.817711 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.817234    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.817711 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.817565    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 28 10:08:14.818128 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.817721    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:14.818128 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.817752    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 28 10:08:14.818128 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.818042    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-z87q9\""
Jul 28 10:08:14.818527 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.818302    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:14.819816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.819796    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:14.820495 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.820317    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 28 10:08:14.820630 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.820610    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-hfjdr\""
Jul 28 10:08:14.820861 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.820832    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.820957 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.820866    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 28 10:08:14.821194 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.821177    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.821992 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.821961    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 28 10:08:14.822410 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.822391    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-jzdx6\""
Jul 28 10:08:14.823856 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.823833    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-p9l6p\""
Jul 28 10:08:14.824476 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.824453    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 28 10:08:14.825889 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.825867    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 28 10:08:14.826066 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.825868    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 28 10:08:14.827841 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.827818    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-systemd-units\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.827937 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.827865    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3bf6896-b4a0-4bf8-b1a3-db1721465614-tmp-dir\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.827937 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.827900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-multus-certs\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.827937 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.827930    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-etc-kubernetes\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.828110 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.827966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.828110 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828103    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.828215 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828139    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysctl-d\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.828215 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828200    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-cni-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.828317 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-run-ovn-kubernetes\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.828317 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-host\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.828317 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828292    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-tuned\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.828466 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-systemd\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.828466 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-ovn\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.828466 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828433    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdp7v\" (UniqueName: \"kubernetes.io/projected/e3bf6896-b4a0-4bf8-b1a3-db1721465614-kube-api-access-mdp7v\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.828616 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-modprobe-d\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.828616 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828499    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-cni-bin\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.828616 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828575    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-daemon-config\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.828616 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.828861 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-hostroot\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.828861 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828666    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-netns\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.828861 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828697    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-cni-multus\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.828861 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828736    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-conf-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.828861 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828769    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-node-log\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.828861 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828808    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-cni-netd\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.828861 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828845    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovnkube-config\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.829219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828884    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-run\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.829219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828915    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-var-lib-kubelet\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.829219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828945    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd2a6033-2fee-4c06-9172-e067fe42263a-cni-binary-copy\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.829219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.828972    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.829219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.829093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dnf76\" (UniqueName: \"kubernetes.io/projected/6699c75e-1572-493b-9107-6e46c03f0241-kube-api-access-dnf76\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.829219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.829130    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-system-cni-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.829219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.829162    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-socket-dir-parent\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.829550 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.829281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxtwx\" (UniqueName: \"kubernetes.io/projected/cd2a6033-2fee-4c06-9172-e067fe42263a-kube-api-access-qxtwx\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.829550 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.829341    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-env-overrides\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.829550 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.829386    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de754c06-e8e4-4e27-ad49-c4d6f1513d34-serviceca\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.829550 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.829433    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9wp2\" (UniqueName: \"kubernetes.io/projected/ddff1dad-46bf-4209-97ff-8252a8347832-kube-api-access-r9wp2\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:14.831264 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.829489    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgtvc\" (UniqueName: \"kubernetes.io/projected/a956fb1b-2c8c-43f0-a5be-2405844edc14-kube-api-access-jgtvc\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.832036 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-system-cni-dir\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.832228 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:14.832446 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832429    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysctl-conf\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.832661 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832607    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a956fb1b-2c8c-43f0-a5be-2405844edc14-tmp\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.832769 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832639    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-run-netns\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.832769 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832709    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.832769 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de754c06-e8e4-4e27-ad49-c4d6f1513d34-host\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.832769 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832763    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-kubernetes\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.832983 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832802    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-lib-modules\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.832983 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-k8s-cni-cncf-io\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.832983 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832854    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-var-lib-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.832983 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832901    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-log-socket\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.832983 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-cni-bin\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.833201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.832989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w88cj\" (UniqueName: \"kubernetes.io/projected/5ad2dd92-dcb7-4fe9-927d-bc213f250105-kube-api-access-w88cj\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.833201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3bf6896-b4a0-4bf8-b1a3-db1721465614-hosts-file\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.833201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74e6f51a-e169-4372-9ba9-48189d368e93-kubelet-config\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.833201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.833201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833121    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-os-release\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.833201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833163    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-kubelet\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.833201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833198    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovnkube-script-lib\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833266    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74e6f51a-e169-4372-9ba9-48189d368e93-dbus\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833290    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-cnibin\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-kubelet\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-slash\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833364    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-etc-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-os-release\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysconfig\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833443    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-systemd\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833481    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-sys\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833511    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovn-node-metrics-cert\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.833555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833536    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrxr\" (UniqueName: \"kubernetes.io/projected/de754c06-e8e4-4e27-ad49-c4d6f1513d34-kube-api-access-nvrxr\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.834204 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833561    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-cnibin\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.834204 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.833586    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.834204 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.834085    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 28 10:08:14.868345 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.868320    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-xczrk"
Jul 28 10:08:14.875810 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.875780    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-xczrk"
Jul 28 10:08:14.914912 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.914884    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 28 10:08:14.933840 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933815    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-host\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.933971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-tuned\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.933971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933881    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-systemd\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.933971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-ovn\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.933971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdp7v\" (UniqueName: \"kubernetes.io/projected/e3bf6896-b4a0-4bf8-b1a3-db1721465614-kube-api-access-mdp7v\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.933971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-host\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.933971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933960    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm7g4\" (UniqueName: \"kubernetes.io/projected/f6f047d6-b659-48d1-a2b3-53e579750620-kube-api-access-qm7g4\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-systemd\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.933989    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-modprobe-d\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-cni-bin\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-daemon-config\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934065    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-hostroot\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934117    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6f047d6-b659-48d1-a2b3-53e579750620-iptables-alerter-script\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934183    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-netns\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-cni-multus\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934257    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 28 10:08:14.934304 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934293    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-ovn\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934235    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-conf-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934348    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-node-log\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934376    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-cni-netd\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovnkube-config\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-device-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-sys-fs\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934481    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-run\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934507    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-var-lib-kubelet\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934534    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd2a6033-2fee-4c06-9172-e067fe42263a-cni-binary-copy\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934566    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934598    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dnf76\" (UniqueName: \"kubernetes.io/projected/6699c75e-1572-493b-9107-6e46c03f0241-kube-api-access-dnf76\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-socket-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934651    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff-agent-certs\") pod \"konnectivity-agent-szf6k\" (UID: \"0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff\") " pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-modprobe-d\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934688    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-system-cni-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934708    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-socket-dir-parent\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.934755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934727    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-conf-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxtwx\" (UniqueName: \"kubernetes.io/projected/cd2a6033-2fee-4c06-9172-e067fe42263a-kube-api-access-qxtwx\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934749    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-env-overrides\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934770    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de754c06-e8e4-4e27-ad49-c4d6f1513d34-serviceca\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934827    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934777    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-node-log\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9wp2\" (UniqueName: \"kubernetes.io/projected/ddff1dad-46bf-4209-97ff-8252a8347832-kube-api-access-r9wp2\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934865    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgtvc\" (UniqueName: \"kubernetes.io/projected/a956fb1b-2c8c-43f0-a5be-2405844edc14-kube-api-access-jgtvc\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934882    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-system-cni-dir\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934926    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysctl-conf\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a956fb1b-2c8c-43f0-a5be-2405844edc14-tmp\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934944    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-var-lib-kubelet\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934963    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-run-netns\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.934984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-cni-netd\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935013    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.935728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935042    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de754c06-e8e4-4e27-ad49-c4d6f1513d34-host\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935069    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-kubernetes\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935097    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-lib-modules\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935121    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-k8s-cni-cncf-io\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-var-lib-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935174    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-log-socket\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935196    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-cni-bin\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w88cj\" (UniqueName: \"kubernetes.io/projected/5ad2dd92-dcb7-4fe9-927d-bc213f250105-kube-api-access-w88cj\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935239    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3bf6896-b4a0-4bf8-b1a3-db1721465614-hosts-file\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935261    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74e6f51a-e169-4372-9ba9-48189d368e93-kubelet-config\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935281    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-os-release\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935329    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-kubelet\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935355    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovnkube-script-lib\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935411    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6f047d6-b659-48d1-a2b3-53e579750620-host-slash\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935438    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74e6f51a-e169-4372-9ba9-48189d368e93-dbus\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.936641 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/cd2a6033-2fee-4c06-9172-e067fe42263a-cni-binary-copy\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935467    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-cnibin\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-kubelet\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935529    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-cni-multus\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935541    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-slash\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935570    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-etc-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935596    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-os-release\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935625    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysconfig\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935625    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovnkube-config\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-systemd\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-sys\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovn-node-metrics-cert\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935726    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-run\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935731    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrxr\" (UniqueName: \"kubernetes.io/projected/de754c06-e8e4-4e27-ad49-c4d6f1513d34-kube-api-access-nvrxr\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-cnibin\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935787    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.937575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-registration-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-lib-modules\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-systemd-units\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-k8s-cni-cncf-io\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-systemd-units\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3bf6896-b4a0-4bf8-b1a3-db1721465614-tmp-dir\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935938    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-var-lib-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935947    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlp5r\" (UniqueName: \"kubernetes.io/projected/2ef48695-9d61-48cd-8a52-7070625394bf-kube-api-access-nlp5r\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.935981    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-log-socket\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936022    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff-konnectivity-ca\") pod \"konnectivity-agent-szf6k\" (UID: \"0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff\") " pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-multus-certs\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936085    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-etc-kubernetes\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/74e6f51a-e169-4372-9ba9-48189d368e93-dbus\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936143    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-cnibin\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936179    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysctl-d\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936192    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-kubelet\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.938441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e3bf6896-b4a0-4bf8-b1a3-db1721465614-tmp-dir\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-cni-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936240    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-var-lib-cni-bin\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-slash\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-cni-bin\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936354    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-os-release\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936395    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-etc-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936408    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysconfig\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936493    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-systemd\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-sys\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936643    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/e3bf6896-b4a0-4bf8-b1a3-db1721465614-hosts-file\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/74e6f51a-e169-4372-9ba9-48189d368e93-kubelet-config\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.936811    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.936926    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret podName:74e6f51a-e169-4372-9ba9-48189d368e93 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:15.436904891 +0000 UTC m=+3.012097824 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret") pod "global-pull-secret-syncer-t6srt" (UID: "74e6f51a-e169-4372-9ba9-48189d368e93") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936941    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-system-cni-dir\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.937018    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:14.942698 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.937067    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs podName:ddff1dad-46bf-4209-97ff-8252a8347832 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:15.437052531 +0000 UTC m=+3.012245454 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs") pod "network-metrics-daemon-vxtwq" (UID: "ddff1dad-46bf-4209-97ff-8252a8347832") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-os-release\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937131    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-hostroot\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937182    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-kubelet\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937183    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysctl-conf\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.936241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-run-ovn-kubernetes\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937399    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-run-ovn-kubernetes\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-netns\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937455    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-run-openvswitch\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937508    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-system-cni-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6699c75e-1572-493b-9107-6e46c03f0241-cnibin\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-socket-dir-parent\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937857    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/de754c06-e8e4-4e27-ad49-c4d6f1513d34-host\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937903    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5ad2dd92-dcb7-4fe9-927d-bc213f250105-host-run-netns\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.937961    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-kubernetes\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.938138    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-cni-dir\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.938185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-etc-kubernetes\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.938315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/cd2a6033-2fee-4c06-9172-e067fe42263a-multus-daemon-config\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.943693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.938333    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-sysctl-d\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.938593    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/cd2a6033-2fee-4c06-9172-e067fe42263a-host-run-multus-certs\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.938971    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/a956fb1b-2c8c-43f0-a5be-2405844edc14-etc-tuned\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.939269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovnkube-script-lib\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.939821    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-cni-binary-copy\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.939877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5ad2dd92-dcb7-4fe9-927d-bc213f250105-env-overrides\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.940211    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/de754c06-e8e4-4e27-ad49-c4d6f1513d34-serviceca\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.940621    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.941035    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/6699c75e-1572-493b-9107-6e46c03f0241-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.944569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.941978    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5ad2dd92-dcb7-4fe9-927d-bc213f250105-ovn-node-metrics-cert\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.946877 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.946855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/a956fb1b-2c8c-43f0-a5be-2405844edc14-tmp\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.949308 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.948921    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:14.949308 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.948947    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:14.949308 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.948969    2560 projected.go:196] Error preparing data for projected volume kube-api-access-twrtv for pod openshift-network-diagnostics/network-check-target-lm85v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:14.949308 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:14.949101    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv podName:a667ef37-25e2-454b-97e2-aa2b32f5b28a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:15.449084455 +0000 UTC m=+3.024277392 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-twrtv" (UniqueName: "kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv") pod "network-check-target-lm85v" (UID: "a667ef37-25e2-454b-97e2-aa2b32f5b28a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:14.950833 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.950458    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w88cj\" (UniqueName: \"kubernetes.io/projected/5ad2dd92-dcb7-4fe9-927d-bc213f250105-kube-api-access-w88cj\") pod \"ovnkube-node-9cmbd\" (UID: \"5ad2dd92-dcb7-4fe9-927d-bc213f250105\") " pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:14.951720 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.951670    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dnf76\" (UniqueName: \"kubernetes.io/projected/6699c75e-1572-493b-9107-6e46c03f0241-kube-api-access-dnf76\") pod \"multus-additional-cni-plugins-2fljt\" (UID: \"6699c75e-1572-493b-9107-6e46c03f0241\") " pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:14.952207 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.952156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxtwx\" (UniqueName: \"kubernetes.io/projected/cd2a6033-2fee-4c06-9172-e067fe42263a-kube-api-access-qxtwx\") pod \"multus-6h4hc\" (UID: \"cd2a6033-2fee-4c06-9172-e067fe42263a\") " pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:14.952298 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.952270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdp7v\" (UniqueName: \"kubernetes.io/projected/e3bf6896-b4a0-4bf8-b1a3-db1721465614-kube-api-access-mdp7v\") pod \"node-resolver-n72pd\" (UID: \"e3bf6896-b4a0-4bf8-b1a3-db1721465614\") " pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:14.952713 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.952695    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9wp2\" (UniqueName: \"kubernetes.io/projected/ddff1dad-46bf-4209-97ff-8252a8347832-kube-api-access-r9wp2\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:14.954052 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.954029    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrxr\" (UniqueName: \"kubernetes.io/projected/de754c06-e8e4-4e27-ad49-c4d6f1513d34-kube-api-access-nvrxr\") pod \"node-ca-6zfzk\" (UID: \"de754c06-e8e4-4e27-ad49-c4d6f1513d34\") " pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:14.956238 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.956213    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgtvc\" (UniqueName: \"kubernetes.io/projected/a956fb1b-2c8c-43f0-a5be-2405844edc14-kube-api-access-jgtvc\") pod \"tuned-t2z2v\" (UID: \"a956fb1b-2c8c-43f0-a5be-2405844edc14\") " pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:14.981205 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.981156    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal" event={"ID":"2cf42ad45c184732bda36fba61d2a3c5","Type":"ContainerStarted","Data":"adba13b24bd810cf0f3f554923c3efb89e403a50b9ba1de02680e1228d34817b"}
Jul 28 10:08:14.983208 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:14.983175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal" event={"ID":"7eb04e870c582d25e244a866d995f954","Type":"ContainerStarted","Data":"ebd811e3a50fd9b5d47182171fabf8f2f139a5c4cba8ae20fb6117977a83f77a"}
Jul 28 10:08:15.039080 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039034    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039191 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039103    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6f047d6-b659-48d1-a2b3-53e579750620-host-slash\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:15.039191 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039136    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-registration-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039191 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlp5r\" (UniqueName: \"kubernetes.io/projected/2ef48695-9d61-48cd-8a52-7070625394bf-kube-api-access-nlp5r\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039194    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff-konnectivity-ca\") pod \"konnectivity-agent-szf6k\" (UID: \"0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff\") " pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:15.039365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039203    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f6f047d6-b659-48d1-a2b3-53e579750620-host-slash\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:15.039365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039227    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm7g4\" (UniqueName: \"kubernetes.io/projected/f6f047d6-b659-48d1-a2b3-53e579750620-kube-api-access-qm7g4\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:15.039365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039256    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6f047d6-b659-48d1-a2b3-53e579750620-iptables-alerter-script\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:15.039365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039284    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-device-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-sys-fs\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039338    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-socket-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039721 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039375    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff-agent-certs\") pod \"konnectivity-agent-szf6k\" (UID: \"0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff\") " pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:15.039721 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039721 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039472    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-registration-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039721 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039536    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-etc-selinux\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039721 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039589    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-device-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.039721 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-sys-fs\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.040069 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-socket-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.040069 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.040030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f6f047d6-b659-48d1-a2b3-53e579750620-iptables-alerter-script\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:15.040069 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.039164    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/2ef48695-9d61-48cd-8a52-7070625394bf-kubelet-dir\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.040635 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.040609    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff-konnectivity-ca\") pod \"konnectivity-agent-szf6k\" (UID: \"0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff\") " pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:15.042574 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.042549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff-agent-certs\") pod \"konnectivity-agent-szf6k\" (UID: \"0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff\") " pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:15.049693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.049646    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlp5r\" (UniqueName: \"kubernetes.io/projected/2ef48695-9d61-48cd-8a52-7070625394bf-kube-api-access-nlp5r\") pod \"aws-ebs-csi-driver-node-8q5vg\" (UID: \"2ef48695-9d61-48cd-8a52-7070625394bf\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.049936 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.049906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm7g4\" (UniqueName: \"kubernetes.io/projected/f6f047d6-b659-48d1-a2b3-53e579750620-kube-api-access-qm7g4\") pod \"iptables-alerter-q2w92\" (UID: \"f6f047d6-b659-48d1-a2b3-53e579750620\") " pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:15.124840 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.124773    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-t2z2v"
Jul 28 10:08:15.136008 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.135979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-6zfzk"
Jul 28 10:08:15.136941 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:15.136905    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda956fb1b_2c8c_43f0_a5be_2405844edc14.slice/crio-b9908d337e68de26d3c9d336a03e8cc0a76baacd90e192b4df4bab5f7ef91299 WatchSource:0}: Error finding container b9908d337e68de26d3c9d336a03e8cc0a76baacd90e192b4df4bab5f7ef91299: Status 404 returned error can't find the container with id b9908d337e68de26d3c9d336a03e8cc0a76baacd90e192b4df4bab5f7ef91299
Jul 28 10:08:15.145070 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.144916    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-6h4hc"
Jul 28 10:08:15.145315 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:15.145287    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde754c06_e8e4_4e27_ad49_c4d6f1513d34.slice/crio-a026847952b35322764322bf3036fdc6dd0afbe82e32e42bb87e37e523f444ea WatchSource:0}: Error finding container a026847952b35322764322bf3036fdc6dd0afbe82e32e42bb87e37e523f444ea: Status 404 returned error can't find the container with id a026847952b35322764322bf3036fdc6dd0afbe82e32e42bb87e37e523f444ea
Jul 28 10:08:15.153084 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.153061    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:15.161288 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.161267    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-n72pd"
Jul 28 10:08:15.162370 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:15.162313    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad2dd92_dcb7_4fe9_927d_bc213f250105.slice/crio-d8900b03ac7c4387b44bbdb075c64efa5a0ce4710dc912334622b7b92edffe8b WatchSource:0}: Error finding container d8900b03ac7c4387b44bbdb075c64efa5a0ce4710dc912334622b7b92edffe8b: Status 404 returned error can't find the container with id d8900b03ac7c4387b44bbdb075c64efa5a0ce4710dc912334622b7b92edffe8b
Jul 28 10:08:15.170734 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:15.170677    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3bf6896_b4a0_4bf8_b1a3_db1721465614.slice/crio-c1427bb71a77a3d0747f0d8fe346e0db29e5e4a36432ec2de9719e3fc35ae76c WatchSource:0}: Error finding container c1427bb71a77a3d0747f0d8fe346e0db29e5e4a36432ec2de9719e3fc35ae76c: Status 404 returned error can't find the container with id c1427bb71a77a3d0747f0d8fe346e0db29e5e4a36432ec2de9719e3fc35ae76c
Jul 28 10:08:15.175625 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.175602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2fljt"
Jul 28 10:08:15.183578 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:15.183555    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6699c75e_1572_493b_9107_6e46c03f0241.slice/crio-961163e04e02f70737a78e73ce22a4a04ea44979b9f11b448986a27ebd431549 WatchSource:0}: Error finding container 961163e04e02f70737a78e73ce22a4a04ea44979b9f11b448986a27ebd431549: Status 404 returned error can't find the container with id 961163e04e02f70737a78e73ce22a4a04ea44979b9f11b448986a27ebd431549
Jul 28 10:08:15.184732 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.184717    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:15.196015 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.195979    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-q2w92"
Jul 28 10:08:15.201688 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.201669    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg"
Jul 28 10:08:15.441954 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.441859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:15.441954 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.441921    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:15.442147 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.441997    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:15.442147 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.442017    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:15.442147 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.442057    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs podName:ddff1dad-46bf-4209-97ff-8252a8347832 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:16.442041274 +0000 UTC m=+4.017234183 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs") pod "network-metrics-daemon-vxtwq" (UID: "ddff1dad-46bf-4209-97ff-8252a8347832") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:15.442147 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.442073    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret podName:74e6f51a-e169-4372-9ba9-48189d368e93 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:16.442064921 +0000 UTC m=+4.017257829 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret") pod "global-pull-secret-syncer-t6srt" (UID: "74e6f51a-e169-4372-9ba9-48189d368e93") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:15.542477 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.542446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:15.542640 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.542615    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:15.542640 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.542640    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:15.542758 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.542651    2560 projected.go:196] Error preparing data for projected volume kube-api-access-twrtv for pod openshift-network-diagnostics/network-check-target-lm85v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:15.542758 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.542713    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv podName:a667ef37-25e2-454b-97e2-aa2b32f5b28a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:16.542697413 +0000 UTC m=+4.117890323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-twrtv" (UniqueName: "kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv") pod "network-check-target-lm85v" (UID: "a667ef37-25e2-454b-97e2-aa2b32f5b28a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:15.878617 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.878540    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:03:15 +0000 UTC" deadline="2028-01-09 00:06:44.665768582 +0000 UTC"
Jul 28 10:08:15.878617 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.878581    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12709h58m28.787192755s"
Jul 28 10:08:15.972831 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.972799    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:15.973001 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:15.972962    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:15.986437 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.986399    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t2z2v" event={"ID":"a956fb1b-2c8c-43f0-a5be-2405844edc14","Type":"ContainerStarted","Data":"b9908d337e68de26d3c9d336a03e8cc0a76baacd90e192b4df4bab5f7ef91299"}
Jul 28 10:08:15.989042 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.989012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fljt" event={"ID":"6699c75e-1572-493b-9107-6e46c03f0241","Type":"ContainerStarted","Data":"961163e04e02f70737a78e73ce22a4a04ea44979b9f11b448986a27ebd431549"}
Jul 28 10:08:15.991215 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.991167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n72pd" event={"ID":"e3bf6896-b4a0-4bf8-b1a3-db1721465614","Type":"ContainerStarted","Data":"c1427bb71a77a3d0747f0d8fe346e0db29e5e4a36432ec2de9719e3fc35ae76c"}
Jul 28 10:08:15.993370 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.993345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"d8900b03ac7c4387b44bbdb075c64efa5a0ce4710dc912334622b7b92edffe8b"}
Jul 28 10:08:15.994701 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.994676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6h4hc" event={"ID":"cd2a6033-2fee-4c06-9172-e067fe42263a","Type":"ContainerStarted","Data":"99aa93d0f46abb23a0a1e73dea2ef085547471274aacc10d651a0f46a673d5bf"}
Jul 28 10:08:15.996054 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:15.996031    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6zfzk" event={"ID":"de754c06-e8e4-4e27-ad49-c4d6f1513d34","Type":"ContainerStarted","Data":"a026847952b35322764322bf3036fdc6dd0afbe82e32e42bb87e37e523f444ea"}
Jul 28 10:08:16.451013 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:16.450969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:16.451219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:16.451081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:16.451300 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.451253    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:16.451357 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.451323    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs podName:ddff1dad-46bf-4209-97ff-8252a8347832 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:18.451303254 +0000 UTC m=+6.026496186 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs") pod "network-metrics-daemon-vxtwq" (UID: "ddff1dad-46bf-4209-97ff-8252a8347832") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:16.451760 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.451729    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:16.451833 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.451813    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret podName:74e6f51a-e169-4372-9ba9-48189d368e93 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:18.451779083 +0000 UTC m=+6.026971999 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret") pod "global-pull-secret-syncer-t6srt" (UID: "74e6f51a-e169-4372-9ba9-48189d368e93") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:16.557272 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:16.552429    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:16.557272 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.552624    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:16.557272 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.552640    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:16.557272 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.552650    2560 projected.go:196] Error preparing data for projected volume kube-api-access-twrtv for pod openshift-network-diagnostics/network-check-target-lm85v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:16.557272 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.552697    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv podName:a667ef37-25e2-454b-97e2-aa2b32f5b28a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:18.55268269 +0000 UTC m=+6.127875598 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-twrtv" (UniqueName: "kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv") pod "network-check-target-lm85v" (UID: "a667ef37-25e2-454b-97e2-aa2b32f5b28a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:16.716872 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:16.716586    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 28 10:08:16.878914 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:16.878870    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-27 10:03:15 +0000 UTC" deadline="2028-04-08 15:08:31.69519616 +0000 UTC"
Jul 28 10:08:16.878914 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:16.878913    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14885h0m14.816288781s"
Jul 28 10:08:16.983858 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:16.981531    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:16.983858 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.981670    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:16.983858 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:16.982065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:16.983858 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:16.982158    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:17.505345 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:17.505309    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6f047d6_b659_48d1_a2b3_53e579750620.slice/crio-bc6c2800e42d2b18e1c1445114401ecd2dd1aba5cc4782358fbb805f02e2aabf WatchSource:0}: Error finding container bc6c2800e42d2b18e1c1445114401ecd2dd1aba5cc4782358fbb805f02e2aabf: Status 404 returned error can't find the container with id bc6c2800e42d2b18e1c1445114401ecd2dd1aba5cc4782358fbb805f02e2aabf
Jul 28 10:08:17.973049 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:17.972962    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:17.973505 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:17.973111    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:18.001682 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:18.001639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg" event={"ID":"2ef48695-9d61-48cd-8a52-7070625394bf","Type":"ContainerStarted","Data":"44b53b7e3c7d4f66256b889172eaed9249a13871bfef9f3cb29d7eaf374deb88"}
Jul 28 10:08:18.002993 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:18.002968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szf6k" event={"ID":"0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff","Type":"ContainerStarted","Data":"6007f32531f941da5bcb8750581714634dbee7e05d220654646addd59ad953f5"}
Jul 28 10:08:18.004553 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:18.004525    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q2w92" event={"ID":"f6f047d6-b659-48d1-a2b3-53e579750620","Type":"ContainerStarted","Data":"bc6c2800e42d2b18e1c1445114401ecd2dd1aba5cc4782358fbb805f02e2aabf"}
Jul 28 10:08:18.470176 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:18.470079    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:18.470176 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:18.470123    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:18.470409 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.470254    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:18.470409 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.470321    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret podName:74e6f51a-e169-4372-9ba9-48189d368e93 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:22.470301885 +0000 UTC m=+10.045494799 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret") pod "global-pull-secret-syncer-t6srt" (UID: "74e6f51a-e169-4372-9ba9-48189d368e93") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:18.470754 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.470653    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:18.470754 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.470725    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs podName:ddff1dad-46bf-4209-97ff-8252a8347832 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:22.470706929 +0000 UTC m=+10.045899852 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs") pod "network-metrics-daemon-vxtwq" (UID: "ddff1dad-46bf-4209-97ff-8252a8347832") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:18.571077 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:18.571035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:18.571290 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.571252    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:18.571290 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.571274    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:18.571290 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.571289    2560 projected.go:196] Error preparing data for projected volume kube-api-access-twrtv for pod openshift-network-diagnostics/network-check-target-lm85v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:18.571469 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.571350    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv podName:a667ef37-25e2-454b-97e2-aa2b32f5b28a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:22.571330852 +0000 UTC m=+10.146523767 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-twrtv" (UniqueName: "kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv") pod "network-check-target-lm85v" (UID: "a667ef37-25e2-454b-97e2-aa2b32f5b28a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:18.972707 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:18.972662    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:18.972916 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.972888    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:18.973021 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:18.972666    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:18.973086 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:18.973016    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:19.973343 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:19.973156    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:19.973343 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:19.973303    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:20.973057 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:20.972856    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:20.973057 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:20.973004    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:20.973462 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:20.973316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:20.973462 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:20.973427    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:21.972444 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:21.972238    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:21.972637 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:21.972535    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:22.018646 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:22.017872    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal" event={"ID":"2cf42ad45c184732bda36fba61d2a3c5","Type":"ContainerStarted","Data":"a3ad5b4542ac503cd688ef03bed90755b93f38cd0fef39c66808377e0e28140e"}
Jul 28 10:08:22.034936 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:22.034237    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-131-51.ec2.internal" podStartSLOduration=8.034214843 podStartE2EDuration="8.034214843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:22.033943138 +0000 UTC m=+9.609136068" watchObservedRunningTime="2026-07-28 10:08:22.034214843 +0000 UTC m=+9.609407783"
Jul 28 10:08:22.508965 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:22.508919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:22.509165 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:22.508983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:22.509165 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.509101    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:22.509165 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.509128    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:22.509320 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.509183    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs podName:ddff1dad-46bf-4209-97ff-8252a8347832 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:30.509161926 +0000 UTC m=+18.084354835 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs") pod "network-metrics-daemon-vxtwq" (UID: "ddff1dad-46bf-4209-97ff-8252a8347832") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:22.509320 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.509201    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret podName:74e6f51a-e169-4372-9ba9-48189d368e93 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:30.509192808 +0000 UTC m=+18.084385719 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret") pod "global-pull-secret-syncer-t6srt" (UID: "74e6f51a-e169-4372-9ba9-48189d368e93") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:22.610056 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:22.609985    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:22.610237 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.610179    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:22.610237 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.610208    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:22.610237 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.610222    2560 projected.go:196] Error preparing data for projected volume kube-api-access-twrtv for pod openshift-network-diagnostics/network-check-target-lm85v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:22.610390 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.610284    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv podName:a667ef37-25e2-454b-97e2-aa2b32f5b28a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:30.610265749 +0000 UTC m=+18.185458669 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-twrtv" (UniqueName: "kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv") pod "network-check-target-lm85v" (UID: "a667ef37-25e2-454b-97e2-aa2b32f5b28a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:22.974710 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:22.974065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:22.974710 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.974196    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:22.975155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:22.975010    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:22.975155 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:22.975116    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:23.972476 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:23.972446    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:23.972960 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:23.972561    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:24.972838 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:24.972807    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:24.973247 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:24.972935    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:24.973247 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:24.973002    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:24.973247 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:24.973121    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:25.972888 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:25.972856    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:25.973309 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:25.972963    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:26.972434 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:26.972397    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:26.972618 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:26.972397    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:26.972618 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:26.972528    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:26.972618 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:26.972567    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:27.972843 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:27.972808    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:27.973274 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:27.972929    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:28.973050 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:28.973016    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:28.973407 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:28.973154    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:28.973407 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:28.973201    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:28.973407 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:28.973311    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:29.972383 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:29.972342    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:29.972631 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:29.972460    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:30.573525 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:30.573488    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:30.574001 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:30.573546    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:30.574001 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.573654    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:30.574001 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.573680    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:30.574001 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.573738    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs podName:ddff1dad-46bf-4209-97ff-8252a8347832 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:46.573705776 +0000 UTC m=+34.148898703 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs") pod "network-metrics-daemon-vxtwq" (UID: "ddff1dad-46bf-4209-97ff-8252a8347832") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:30.574001 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.573754    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret podName:74e6f51a-e169-4372-9ba9-48189d368e93 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:46.573747228 +0000 UTC m=+34.148940137 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret") pod "global-pull-secret-syncer-t6srt" (UID: "74e6f51a-e169-4372-9ba9-48189d368e93") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:30.674224 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:30.674190    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:30.674410 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.674325    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:30.674410 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.674345    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:30.674410 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.674358    2560 projected.go:196] Error preparing data for projected volume kube-api-access-twrtv for pod openshift-network-diagnostics/network-check-target-lm85v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:30.674576 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.674416    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv podName:a667ef37-25e2-454b-97e2-aa2b32f5b28a nodeName:}" failed. No retries permitted until 2026-07-28 10:08:46.674399687 +0000 UTC m=+34.249592597 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-twrtv" (UniqueName: "kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv") pod "network-check-target-lm85v" (UID: "a667ef37-25e2-454b-97e2-aa2b32f5b28a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:30.972755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:30.972649    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:30.972946 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.972796    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:30.972946 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:30.972855    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:30.973060 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:30.972996    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:31.973099 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:31.973064    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:31.973501 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:31.973179    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:32.972733 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:32.972692    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:32.972880 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:32.972811    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:32.972880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:32.972868    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:32.973091 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:32.973061    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:33.972320 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:33.972297    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:33.972740 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:33.972406    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:34.045760 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:34.045113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-t2z2v" event={"ID":"a956fb1b-2c8c-43f0-a5be-2405844edc14","Type":"ContainerStarted","Data":"26e3131e0c72c51484fd827e4e2e6f246f105ac26eb0646fcca7be72f0dd7551"}
Jul 28 10:08:34.066441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:34.066404    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"78091c17829d8cb711bbe531657d0cc5d09d150a1f43c7a936afa36930008293"}
Jul 28 10:08:34.066727 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:34.066670    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-t2z2v" podStartSLOduration=2.411358998 podStartE2EDuration="21.066651772s" totalImagesPullingTime="18.655292774s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:34.066268706 +0000 UTC m=+21.641461637" watchObservedRunningTime="2026-07-28 10:08:34.066651772 +0000 UTC m=+21.641844701"
Jul 28 10:08:34.069572 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:34.069540    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-6h4hc" event={"ID":"cd2a6033-2fee-4c06-9172-e067fe42263a","Type":"ContainerStarted","Data":"441f7c9489fdf3e0f854cd4781de6b8966862b2892d8fd45613b6222dbedad96"}
Jul 28 10:08:34.972821 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:34.972788    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:34.973532 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:34.972795    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:34.973532 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:34.972927    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:34.973532 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:34.972993    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:35.072598 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.072572    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-n72pd" event={"ID":"e3bf6896-b4a0-4bf8-b1a3-db1721465614","Type":"ContainerStarted","Data":"2344656b31da2df7f129412ddbf326423a1f2c45f0979a89cc9a44f5f8d4b580"}
Jul 28 10:08:35.075215 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.075194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"7ccc2a3d745383cca2ff4278b0ce0f8157f10ae7cd412f2b0472842b18329203"}
Jul 28 10:08:35.075311 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.075265    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"29dfe4cbf5b6eb89ae0452b70cb555fbefd8d32c6f3c35c5bec54116755be4eb"}
Jul 28 10:08:35.075373 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.075323    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"8e7775d449e0c96ed9a347a52fcf65fffc3d7030c2848a64a90234e864c7efb0"}
Jul 28 10:08:35.075412 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.075371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"d0fbf0cede6d4b1a7d9a508f20ab21d8d22e5bcf20f6fcfc7061dbebcabfc138"}
Jul 28 10:08:35.075412 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.075408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"292286ddac672d50cdf60533e987114946f5e2c29a2eabe114f07cd876befa39"}
Jul 28 10:08:35.076373 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.076353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-6zfzk" event={"ID":"de754c06-e8e4-4e27-ad49-c4d6f1513d34","Type":"ContainerStarted","Data":"0595e2c247b737fa8080fd40d006f3ecaa4fc9a7e9a98615d35acbc2090aa82b"}
Jul 28 10:08:35.077766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.077740    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg" event={"ID":"2ef48695-9d61-48cd-8a52-7070625394bf","Type":"ContainerStarted","Data":"9e1b988e673be4d462021ae0ea91441786cc643781fa98408813012639766251"}
Jul 28 10:08:35.079122 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.079101    2560 generic.go:356] "Generic (PLEG): container finished" podID="6699c75e-1572-493b-9107-6e46c03f0241" containerID="334e971acb80311b9b51d49d96e4a20ca7084742dc85e1c05bbc4b7cba7bf357" exitCode=0
Jul 28 10:08:35.079215 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.079179    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fljt" event={"ID":"6699c75e-1572-493b-9107-6e46c03f0241","Type":"ContainerDied","Data":"334e971acb80311b9b51d49d96e4a20ca7084742dc85e1c05bbc4b7cba7bf357"}
Jul 28 10:08:35.080620 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.080595    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-szf6k" event={"ID":"0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff","Type":"ContainerStarted","Data":"8c0936e4c9bb76e271476b73daf1ba8a1d0e52ce7e51165da5ac63d6377435db"}
Jul 28 10:08:35.081865 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.081843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-q2w92" event={"ID":"f6f047d6-b659-48d1-a2b3-53e579750620","Type":"ContainerStarted","Data":"b412d49d0858ff4c053b989c5a59460b10b702af7033d0c38992fb4a3dc0332c"}
Jul 28 10:08:35.083131 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.083055    2560 generic.go:356] "Generic (PLEG): container finished" podID="7eb04e870c582d25e244a866d995f954" containerID="f1f9a46c5183783955dc1bf9244d9d78c82f2421e31b0f03223cad8a2b164cd5" exitCode=0
Jul 28 10:08:35.083131 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.083121    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal" event={"ID":"7eb04e870c582d25e244a866d995f954","Type":"ContainerDied","Data":"f1f9a46c5183783955dc1bf9244d9d78c82f2421e31b0f03223cad8a2b164cd5"}
Jul 28 10:08:35.090885 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.090853    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-n72pd" podStartSLOduration=8.483888123 podStartE2EDuration="22.090841631s" totalImagesPullingTime="13.606953508s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:35.090655725 +0000 UTC m=+22.665848654" watchObservedRunningTime="2026-07-28 10:08:35.090841631 +0000 UTC m=+22.666034561"
Jul 28 10:08:35.091097 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.091080    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-6h4hc" podStartSLOduration=3.455124257 podStartE2EDuration="22.091076099s" totalImagesPullingTime="18.635951842s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:34.089637017 +0000 UTC m=+21.664829946" watchObservedRunningTime="2026-07-28 10:08:35.091076099 +0000 UTC m=+22.666269027"
Jul 28 10:08:35.106265 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.106099    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-6zfzk" podStartSLOduration=15.838957802 podStartE2EDuration="22.10608758s" totalImagesPullingTime="6.267129778s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:35.105809516 +0000 UTC m=+22.681002446" watchObservedRunningTime="2026-07-28 10:08:35.10608758 +0000 UTC m=+22.681280509"
Jul 28 10:08:35.158007 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.157956    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-szf6k" podStartSLOduration=5.836493067 podStartE2EDuration="22.157938734s" totalImagesPullingTime="16.321445667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:35.157704369 +0000 UTC m=+22.732897300" watchObservedRunningTime="2026-07-28 10:08:35.157938734 +0000 UTC m=+22.733131662"
Jul 28 10:08:35.196909 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.196887    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 28 10:08:35.892871 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.892772    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-28T10:08:35.196902468Z","UUID":"0d124f90-a311-48cb-b220-28c2d768d0d4","Handler":null,"Name":"","Endpoint":""}
Jul 28 10:08:35.894395 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.894375    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 28 10:08:35.894514 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.894422    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 28 10:08:35.973130 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:35.973102    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:35.973537 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:35.973217    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:36.087597 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:36.087567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg" event={"ID":"2ef48695-9d61-48cd-8a52-7070625394bf","Type":"ContainerStarted","Data":"29ec361eee77df5a166e6211128ef71d9ca961df1c3f0b1bd36f62aef1d8b852"}
Jul 28 10:08:36.089356 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:36.089320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal" event={"ID":"7eb04e870c582d25e244a866d995f954","Type":"ContainerStarted","Data":"3444c6a23753fcbb7fd330afe6405c2e020a95553749cfb0430796b851d0ba15"}
Jul 28 10:08:36.106689 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:36.106649    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-131-51.ec2.internal" podStartSLOduration=22.106631967 podStartE2EDuration="22.106631967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:36.106146245 +0000 UTC m=+23.681339174" watchObservedRunningTime="2026-07-28 10:08:36.106631967 +0000 UTC m=+23.681824897"
Jul 28 10:08:36.125022 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:36.124986    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-q2w92" podStartSLOduration=15.67330013 podStartE2EDuration="23.124970962s" totalImagesPullingTime="7.451670832s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:36.124552774 +0000 UTC m=+23.699745704" watchObservedRunningTime="2026-07-28 10:08:36.124970962 +0000 UTC m=+23.700163894"
Jul 28 10:08:36.821203 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:36.821171    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:36.821832 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:36.821813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:36.977837 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:36.977811    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:36.977837 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:36.977825    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:36.978338 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:36.977934    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:36.978338 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:36.978063    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:37.098290 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:37.098253    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"c4f469d7fc76bfe20491d02b4017fe308445dbfad98bc7a700e093915b880a11"}
Jul 28 10:08:37.100424 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:37.100396    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg" event={"ID":"2ef48695-9d61-48cd-8a52-7070625394bf","Type":"ContainerStarted","Data":"4d8b90ebaba7af07a2140f54f6d12bf1fbd8821e21256c3771904295e8552b04"}
Jul 28 10:08:37.100915 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:37.100893    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:37.101290 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:37.101271    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-szf6k"
Jul 28 10:08:37.120675 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:37.120624    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-8q5vg" podStartSLOduration=5.428259802 podStartE2EDuration="24.120612248s" totalImagesPullingTime="18.692352446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:37.120444744 +0000 UTC m=+24.695637674" watchObservedRunningTime="2026-07-28 10:08:37.120612248 +0000 UTC m=+24.695805184"
Jul 28 10:08:37.972526 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:37.972497    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:37.972692 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:37.972605    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:38.976347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:38.976319    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:38.976770 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:38.976327    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:38.976770 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:38.976447    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:38.976770 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:38.976502    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:39.972299 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:39.972140    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:39.972416 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:39.972392    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:40.110520 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:40.110480    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" event={"ID":"5ad2dd92-dcb7-4fe9-927d-bc213f250105","Type":"ContainerStarted","Data":"e45d12a347649d8e45de26ef6931db144bccc797ced770f0520045ce653c8cd2"}
Jul 28 10:08:40.111380 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:40.110676    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:40.112597 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:40.112572    2560 generic.go:356] "Generic (PLEG): container finished" podID="6699c75e-1572-493b-9107-6e46c03f0241" containerID="a2c1ddf534137c38f012c3fb436e76a0eb3884006c1acfa724e28a17afa9bc48" exitCode=0
Jul 28 10:08:40.112691 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:40.112602    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fljt" event={"ID":"6699c75e-1572-493b-9107-6e46c03f0241","Type":"ContainerDied","Data":"a2c1ddf534137c38f012c3fb436e76a0eb3884006c1acfa724e28a17afa9bc48"}
Jul 28 10:08:40.125998 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:40.125978    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:40.138990 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:40.138890    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd" podStartSLOduration=8.413071601 podStartE2EDuration="27.138873003s" totalImagesPullingTime="18.725801402s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:40.136228672 +0000 UTC m=+27.711421596" watchObservedRunningTime="2026-07-28 10:08:40.138873003 +0000 UTC m=+27.714065931"
Jul 28 10:08:40.973851 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:40.973545    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:40.974097 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:40.973915    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:40.974097 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:40.973982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:40.974097 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:40.974070    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:41.115089 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.115054    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:41.115519 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.115138    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:41.132666 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.132642    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:08:41.678590 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.678501    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxtwq"]
Jul 28 10:08:41.678772 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.678629    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:41.678846 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:41.678785    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:41.681952 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.681913    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t6srt"]
Jul 28 10:08:41.682088 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.682013    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:41.682137 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:41.682114    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:41.682492 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.682466    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lm85v"]
Jul 28 10:08:41.682597 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:41.682568    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:41.682678 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:41.682658    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:42.118748 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:42.118650    2560 generic.go:356] "Generic (PLEG): container finished" podID="6699c75e-1572-493b-9107-6e46c03f0241" containerID="8e044f69d940e2063b86dda662b0868fdef17a0b036e445efd0a3f1cc58adaf9" exitCode=0
Jul 28 10:08:42.119120 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:42.118750    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fljt" event={"ID":"6699c75e-1572-493b-9107-6e46c03f0241","Type":"ContainerDied","Data":"8e044f69d940e2063b86dda662b0868fdef17a0b036e445efd0a3f1cc58adaf9"}
Jul 28 10:08:42.973731 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:42.973642    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:42.973869 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:42.973728    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:42.973869 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:42.973748    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:42.973869 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:42.973808    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:43.124007 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:43.123971    2560 generic.go:356] "Generic (PLEG): container finished" podID="6699c75e-1572-493b-9107-6e46c03f0241" containerID="5eef404ad69853c617589c4f4f34fe41907f12e931a306ca9415f912c3e9c444" exitCode=0
Jul 28 10:08:43.124432 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:43.124019    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fljt" event={"ID":"6699c75e-1572-493b-9107-6e46c03f0241","Type":"ContainerDied","Data":"5eef404ad69853c617589c4f4f34fe41907f12e931a306ca9415f912c3e9c444"}
Jul 28 10:08:43.972420 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:43.972341    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:43.972568 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:43.972484    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:44.972663 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:44.972634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:44.972663 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:44.972671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:44.973290 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:44.972778    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-t6srt" podUID="74e6f51a-e169-4372-9ba9-48189d368e93"
Jul 28 10:08:44.973290 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:44.972922    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-lm85v" podUID="a667ef37-25e2-454b-97e2-aa2b32f5b28a"
Jul 28 10:08:45.972663 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:45.972633    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:45.972860 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:45.972774    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-vxtwq" podUID="ddff1dad-46bf-4209-97ff-8252a8347832"
Jul 28 10:08:46.571486 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.571309    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-131-51.ec2.internal" event="NodeReady"
Jul 28 10:08:46.571669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.571597    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 28 10:08:46.593696 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.593616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:46.593696 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.593652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:46.593930 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.593785    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:46.593930 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.593838    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret podName:74e6f51a-e169-4372-9ba9-48189d368e93 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:18.593822173 +0000 UTC m=+66.169015080 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret") pod "global-pull-secret-syncer-t6srt" (UID: "74e6f51a-e169-4372-9ba9-48189d368e93") : object "kube-system"/"original-pull-secret" not registered
Jul 28 10:08:46.593930 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.593785    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:46.593930 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.593930    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs podName:ddff1dad-46bf-4209-97ff-8252a8347832 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:18.59391576 +0000 UTC m=+66.169108671 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs") pod "network-metrics-daemon-vxtwq" (UID: "ddff1dad-46bf-4209-97ff-8252a8347832") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 28 10:08:46.627871 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.627833    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-4gf6b"]
Jul 28 10:08:46.649203 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.649175    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-vswsn"]
Jul 28 10:08:46.649382 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.649345    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.651729 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.651694    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 28 10:08:46.651863 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.651768    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 28 10:08:46.651863 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.651780    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kvgz4\""
Jul 28 10:08:46.661348 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.661322    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vswsn"]
Jul 28 10:08:46.661459 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.661360    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4gf6b"]
Jul 28 10:08:46.661459 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.661427    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vswsn"
Jul 28 10:08:46.665210 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.664140    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 28 10:08:46.665821 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.665797    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 28 10:08:46.665821 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.665808    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7qffd\""
Jul 28 10:08:46.665821 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.665818    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 28 10:08:46.694029 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.694008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:46.694212 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.694196    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 28 10:08:46.694286 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.694220    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 28 10:08:46.694286 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.694233    2560 projected.go:196] Error preparing data for projected volume kube-api-access-twrtv for pod openshift-network-diagnostics/network-check-target-lm85v: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:46.694389 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.694305    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv podName:a667ef37-25e2-454b-97e2-aa2b32f5b28a nodeName:}" failed. No retries permitted until 2026-07-28 10:09:18.694286823 +0000 UTC m=+66.269479746 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-twrtv" (UniqueName: "kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv") pod "network-check-target-lm85v" (UID: "a667ef37-25e2-454b-97e2-aa2b32f5b28a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 28 10:08:46.794812 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.794774    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.794812 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.794810    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jldw6\" (UniqueName: \"kubernetes.io/projected/fda1e514-a122-4d60-8ba6-564837522a48-kube-api-access-jldw6\") pod \"ingress-canary-vswsn\" (UID: \"fda1e514-a122-4d60-8ba6-564837522a48\") " pod="openshift-ingress-canary/ingress-canary-vswsn"
Jul 28 10:08:46.795060 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.794851    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fda1e514-a122-4d60-8ba6-564837522a48-cert\") pod \"ingress-canary-vswsn\" (UID: \"fda1e514-a122-4d60-8ba6-564837522a48\") " pod="openshift-ingress-canary/ingress-canary-vswsn"
Jul 28 10:08:46.795060 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.794918    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-config-volume\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.795060 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.794973    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-tmp-dir\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.795060 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.794996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltrtf\" (UniqueName: \"kubernetes.io/projected/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-kube-api-access-ltrtf\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.895853 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.895822    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-config-volume\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.896032 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.895892    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-tmp-dir\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.896032 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.895917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltrtf\" (UniqueName: \"kubernetes.io/projected/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-kube-api-access-ltrtf\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.896032 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.895938    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.896032 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.895957    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jldw6\" (UniqueName: \"kubernetes.io/projected/fda1e514-a122-4d60-8ba6-564837522a48-kube-api-access-jldw6\") pod \"ingress-canary-vswsn\" (UID: \"fda1e514-a122-4d60-8ba6-564837522a48\") " pod="openshift-ingress-canary/ingress-canary-vswsn"
Jul 28 10:08:46.896032 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.895994    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fda1e514-a122-4d60-8ba6-564837522a48-cert\") pod \"ingress-canary-vswsn\" (UID: \"fda1e514-a122-4d60-8ba6-564837522a48\") " pod="openshift-ingress-canary/ingress-canary-vswsn"
Jul 28 10:08:46.896287 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.896087    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:46.896287 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:46.896170    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls podName:1416cd8c-7960-4cc9-8444-bc3ad41e74d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:47.396149485 +0000 UTC m=+34.971342393 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls") pod "dns-default-4gf6b" (UID: "1416cd8c-7960-4cc9-8444-bc3ad41e74d2") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:46.896401 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.896303    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-tmp-dir\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.896502 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.896484    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-config-volume\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.900642 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.900620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/fda1e514-a122-4d60-8ba6-564837522a48-cert\") pod \"ingress-canary-vswsn\" (UID: \"fda1e514-a122-4d60-8ba6-564837522a48\") " pod="openshift-ingress-canary/ingress-canary-vswsn"
Jul 28 10:08:46.909171 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.909151    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jldw6\" (UniqueName: \"kubernetes.io/projected/fda1e514-a122-4d60-8ba6-564837522a48-kube-api-access-jldw6\") pod \"ingress-canary-vswsn\" (UID: \"fda1e514-a122-4d60-8ba6-564837522a48\") " pod="openshift-ingress-canary/ingress-canary-vswsn"
Jul 28 10:08:46.909283 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.909169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltrtf\" (UniqueName: \"kubernetes.io/projected/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-kube-api-access-ltrtf\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:46.972347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.972314    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:08:46.972562 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.972352    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:08:46.974416 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.974392    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-vswsn"
Jul 28 10:08:46.975091 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.975071    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:08:46.975201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.975112    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nwnjm\""
Jul 28 10:08:46.975201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.975170    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:08:46.975201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:46.975177    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:08:47.400520 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:47.400495    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:47.400688 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:47.400623    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:47.400688 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:47.400677    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls podName:1416cd8c-7960-4cc9-8444-bc3ad41e74d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:48.400664937 +0000 UTC m=+35.975857844 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls") pod "dns-default-4gf6b" (UID: "1416cd8c-7960-4cc9-8444-bc3ad41e74d2") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:47.545465 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:47.545426    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-vswsn"]
Jul 28 10:08:47.603850 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:47.603816    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda1e514_a122_4d60_8ba6_564837522a48.slice/crio-dd49a9c400c49f72759ef242fce8c65cb20c88883d908bddafd35092e62b9880 WatchSource:0}: Error finding container dd49a9c400c49f72759ef242fce8c65cb20c88883d908bddafd35092e62b9880: Status 404 returned error can't find the container with id dd49a9c400c49f72759ef242fce8c65cb20c88883d908bddafd35092e62b9880
Jul 28 10:08:47.972658 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:47.972426    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:08:47.974995 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:47.974977    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:08:47.975574 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:47.975021    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgfjj\""
Jul 28 10:08:48.137783 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:48.137747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vswsn" event={"ID":"fda1e514-a122-4d60-8ba6-564837522a48","Type":"ContainerStarted","Data":"dd49a9c400c49f72759ef242fce8c65cb20c88883d908bddafd35092e62b9880"}
Jul 28 10:08:48.140991 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:48.140958    2560 generic.go:356] "Generic (PLEG): container finished" podID="6699c75e-1572-493b-9107-6e46c03f0241" containerID="3ff1240a663c09e653bbdc30c3f5af78a037e30468682e306e3d5e8758215727" exitCode=0
Jul 28 10:08:48.141145 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:48.140990    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fljt" event={"ID":"6699c75e-1572-493b-9107-6e46c03f0241","Type":"ContainerDied","Data":"3ff1240a663c09e653bbdc30c3f5af78a037e30468682e306e3d5e8758215727"}
Jul 28 10:08:48.407963 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:48.407925    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:48.408129 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:48.408062    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:48.408129 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:48.408126    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls podName:1416cd8c-7960-4cc9-8444-bc3ad41e74d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:50.408107453 +0000 UTC m=+37.983300360 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls") pod "dns-default-4gf6b" (UID: "1416cd8c-7960-4cc9-8444-bc3ad41e74d2") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:49.146971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:49.146933    2560 generic.go:356] "Generic (PLEG): container finished" podID="6699c75e-1572-493b-9107-6e46c03f0241" containerID="d54a5c26b82cf0e7133cdcac46474f642e486f436e392decc28cd645510ca31e" exitCode=0
Jul 28 10:08:49.147443 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:49.146980    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fljt" event={"ID":"6699c75e-1572-493b-9107-6e46c03f0241","Type":"ContainerDied","Data":"d54a5c26b82cf0e7133cdcac46474f642e486f436e392decc28cd645510ca31e"}
Jul 28 10:08:50.153791 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.153747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2fljt" event={"ID":"6699c75e-1572-493b-9107-6e46c03f0241","Type":"ContainerStarted","Data":"b12d58e7b7521263f988e9a697874f9282a0baa7e4a9cff31b0eb7c2892e2975"}
Jul 28 10:08:50.155300 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.155263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-vswsn" event={"ID":"fda1e514-a122-4d60-8ba6-564837522a48","Type":"ContainerStarted","Data":"606377144e1c4c2c9faa15c929a0666835fff3934a7a32e5b29c881d7e0ca49d"}
Jul 28 10:08:50.189100 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.189050    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2fljt" podStartSLOduration=12.395628862 podStartE2EDuration="37.189032832s" totalImagesPullingTime="24.79340397s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.187136086 +0000 UTC m=+37.762329016" watchObservedRunningTime="2026-07-28 10:08:50.189032832 +0000 UTC m=+37.764225762"
Jul 28 10:08:50.190178 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.190152    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-gk59v"]
Jul 28 10:08:50.199501 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.199481    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.201761 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.201736    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 28 10:08:50.201872 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.201736    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vsm7x\""
Jul 28 10:08:50.201872 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.201782    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 28 10:08:50.201872 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.201708    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 28 10:08:50.201872 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.201856    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 28 10:08:50.206130 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.206012    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gk59v"]
Jul 28 10:08:50.208286 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.208152    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-vswsn" podStartSLOduration=1.817755176 podStartE2EDuration="4.20813642s" totalImagesPullingTime="2.390381244s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:08:50.20746855 +0000 UTC m=+37.782661480" watchObservedRunningTime="2026-07-28 10:08:50.20813642 +0000 UTC m=+37.783329350"
Jul 28 10:08:50.323406 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.323332    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39fdcf2e-356b-49df-8e08-10a12febca9e-crio-socket\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.323572 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.323461    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39fdcf2e-356b-49df-8e08-10a12febca9e-data-volume\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.323572 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.323490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.323685 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.323623    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xv8td\" (UniqueName: \"kubernetes.io/projected/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-api-access-xv8td\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.323685 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.323663    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.424828 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.424793    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xv8td\" (UniqueName: \"kubernetes.io/projected/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-api-access-xv8td\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.424828 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.424829    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.425066 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.424851    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:50.425066 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.424931    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:50.425066 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.424949    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:50.425066 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.424983    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39fdcf2e-356b-49df-8e08-10a12febca9e-crio-socket\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.425066 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.424997    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls podName:1416cd8c-7960-4cc9-8444-bc3ad41e74d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:54.424980409 +0000 UTC m=+42.000173315 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls") pod "dns-default-4gf6b" (UID: "1416cd8c-7960-4cc9-8444-bc3ad41e74d2") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:50.425066 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.425033    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:50.925012962 +0000 UTC m=+38.500205872 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:50.425330 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.425091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39fdcf2e-356b-49df-8e08-10a12febca9e-data-volume\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.425330 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.425112    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.425330 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.425200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/39fdcf2e-356b-49df-8e08-10a12febca9e-crio-socket\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.425330 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.425220    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:50.425330 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.425273    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:50.925261777 +0000 UTC m=+38.500454686 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:50.425480 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.425414    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/39fdcf2e-356b-49df-8e08-10a12febca9e-data-volume\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.434292 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.434261    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xv8td\" (UniqueName: \"kubernetes.io/projected/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-api-access-xv8td\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.928878 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.928843    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.929042 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:50.928933    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:50.929042 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.928986    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:50.929042 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.929006    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:50.929341 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.929318    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:51.929238387 +0000 UTC m=+39.504431298 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:50.929405 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:50.929354    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:51.929341784 +0000 UTC m=+39.504534691 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:51.934618 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:51.934464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:51.934618 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:51.934603    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:51.935105 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:51.934645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:51.935105 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:51.934667    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:53.93465034 +0000 UTC m=+41.509843246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:51.935105 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:51.934737    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:51.935105 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:51.934780    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:53.934769173 +0000 UTC m=+41.509962080 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:52.458509 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:52.458482    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n72pd_e3bf6896-b4a0-4bf8-b1a3-db1721465614/dns-node-resolver/0.log"
Jul 28 10:08:52.858004 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:52.857936    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6zfzk_de754c06-e8e4-4e27-ad49-c4d6f1513d34/node-ca/0.log"
Jul 28 10:08:53.951801 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:53.951759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:53.952199 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:53.951820    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:53.952199 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:53.951895    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:53.952199 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:53.951940    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:53.952199 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:53.951957    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:57.951940873 +0000 UTC m=+45.527133779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:53.952199 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:53.951980    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:08:57.951968174 +0000 UTC m=+45.527161080 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:54.058526 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:54.058497    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vswsn_fda1e514-a122-4d60-8ba6-564837522a48/serve-healthcheck-canary/0.log"
Jul 28 10:08:54.454995 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:54.454961    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:08:54.455170 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:54.455118    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:08:54.455217 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:54.455182    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls podName:1416cd8c-7960-4cc9-8444-bc3ad41e74d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:02.455167061 +0000 UTC m=+50.030359968 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls") pod "dns-default-4gf6b" (UID: "1416cd8c-7960-4cc9-8444-bc3ad41e74d2") : secret "dns-default-metrics-tls" not found
Jul 28 10:08:55.461394 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.461358    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz"]
Jul 28 10:08:55.492906 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.492878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz"]
Jul 28 10:08:55.493057 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.492968    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz"
Jul 28 10:08:55.495646 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.495626    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 28 10:08:55.495747 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.495645    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:08:55.495790 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.495758    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-stqtr\""
Jul 28 10:08:55.561534 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.561509    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"]
Jul 28 10:08:55.583071 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.583044    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"]
Jul 28 10:08:55.583227 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.583162    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.585277 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.585246    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-r98cc\""
Jul 28 10:08:55.585402 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.585253    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 28 10:08:55.585402 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.585347    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 28 10:08:55.585505 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.585420    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:08:55.585853 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.585837    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 28 10:08:55.595411 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.595393    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-5568q"]
Jul 28 10:08:55.595516 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.595502    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.597584 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.597552    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 28 10:08:55.597584 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.597573    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-llvb4\""
Jul 28 10:08:55.597760 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.597551    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:08:55.597812 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.597772    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 28 10:08:55.597870 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.597853    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 28 10:08:55.614059 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.614039    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"]
Jul 28 10:08:55.614132 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.614074    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"]
Jul 28 10:08:55.614132 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.614093    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-5568q"]
Jul 28 10:08:55.614213 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.614144    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.616055 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.616040    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 28 10:08:55.616265 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.616249    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 28 10:08:55.616340 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.616249    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-9997t\""
Jul 28 10:08:55.616403 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.616389    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 28 10:08:55.616454 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.616404    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:08:55.620526 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.620510    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 28 10:08:55.663221 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.663188    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"]
Jul 28 10:08:55.663441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.663422    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nhc7\" (UniqueName: \"kubernetes.io/projected/ae99e605-6566-498c-87dc-a3c0c2bf3fbe-kube-api-access-9nhc7\") pod \"volume-data-source-validator-549c6d54cf-sg5vz\" (UID: \"ae99e605-6566-498c-87dc-a3c0c2bf3fbe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz"
Jul 28 10:08:55.676590 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.676565    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"]
Jul 28 10:08:55.676708 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.676655    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:55.679198 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.679176    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 28 10:08:55.679438 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.679240    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 28 10:08:55.679438 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.679350    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-pf8pw\""
Jul 28 10:08:55.679438 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.679384    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 28 10:08:55.764265 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764176    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jcnwx\" (UniqueName: \"kubernetes.io/projected/ae3e4780-f446-480b-917b-d6d0aa02cfa8-kube-api-access-jcnwx\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.764265 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764215    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecc8553b-b5a2-482e-bb37-6281729826a2-config\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.764265 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764232    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-459n2\" (UniqueName: \"kubernetes.io/projected/ecc8553b-b5a2-482e-bb37-6281729826a2-kube-api-access-459n2\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.764531 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764311    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae3e4780-f446-480b-917b-d6d0aa02cfa8-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.764531 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764347    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fab0b7f4-41fe-4684-b4fe-5497dfe59961-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.764531 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764389    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxfk7\" (UniqueName: \"kubernetes.io/projected/fab0b7f4-41fe-4684-b4fe-5497dfe59961-kube-api-access-hxfk7\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.764531 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764465    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc8553b-b5a2-482e-bb37-6281729826a2-trusted-ca\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.764745 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764535    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nhc7\" (UniqueName: \"kubernetes.io/projected/ae99e605-6566-498c-87dc-a3c0c2bf3fbe-kube-api-access-9nhc7\") pod \"volume-data-source-validator-549c6d54cf-sg5vz\" (UID: \"ae99e605-6566-498c-87dc-a3c0c2bf3fbe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz"
Jul 28 10:08:55.764745 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764574    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecc8553b-b5a2-482e-bb37-6281729826a2-serving-cert\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.764745 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764599    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae3e4780-f446-480b-917b-d6d0aa02cfa8-config\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.764745 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.764626    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fab0b7f4-41fe-4684-b4fe-5497dfe59961-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.774825 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.774805    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nhc7\" (UniqueName: \"kubernetes.io/projected/ae99e605-6566-498c-87dc-a3c0c2bf3fbe-kube-api-access-9nhc7\") pod \"volume-data-source-validator-549c6d54cf-sg5vz\" (UID: \"ae99e605-6566-498c-87dc-a3c0c2bf3fbe\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz"
Jul 28 10:08:55.802804 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.802779    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz"
Jul 28 10:08:55.864999 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.864969    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecc8553b-b5a2-482e-bb37-6281729826a2-serving-cert\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.864999 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae3e4780-f446-480b-917b-d6d0aa02cfa8-config\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.865185 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865026    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fab0b7f4-41fe-4684-b4fe-5497dfe59961-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.865185 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865066    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jcnwx\" (UniqueName: \"kubernetes.io/projected/ae3e4780-f446-480b-917b-d6d0aa02cfa8-kube-api-access-jcnwx\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.865185 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865092    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecc8553b-b5a2-482e-bb37-6281729826a2-config\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.865185 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865115    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-459n2\" (UniqueName: \"kubernetes.io/projected/ecc8553b-b5a2-482e-bb37-6281729826a2-kube-api-access-459n2\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.865185 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865144    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:55.865907 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865209    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae3e4780-f446-480b-917b-d6d0aa02cfa8-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.865907 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865252    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fab0b7f4-41fe-4684-b4fe-5497dfe59961-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.865907 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865277    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxfk7\" (UniqueName: \"kubernetes.io/projected/fab0b7f4-41fe-4684-b4fe-5497dfe59961-kube-api-access-hxfk7\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.865907 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865323    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc8553b-b5a2-482e-bb37-6281729826a2-trusted-ca\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.865907 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.865369    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m24lj\" (UniqueName: \"kubernetes.io/projected/492fda93-ef2b-4cb0-88c5-88e9598bc195-kube-api-access-m24lj\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:55.871111 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.871084    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ae3e4780-f446-480b-917b-d6d0aa02cfa8-config\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.872431 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.872037    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ae3e4780-f446-480b-917b-d6d0aa02cfa8-serving-cert\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.872431 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.872169    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/ecc8553b-b5a2-482e-bb37-6281729826a2-trusted-ca\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.872431 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.872405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ecc8553b-b5a2-482e-bb37-6281729826a2-serving-cert\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.885816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.876676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ecc8553b-b5a2-482e-bb37-6281729826a2-config\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.885816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.877269    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/fab0b7f4-41fe-4684-b4fe-5497dfe59961-config\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.885816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.880697    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/fab0b7f4-41fe-4684-b4fe-5497dfe59961-serving-cert\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.885816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.883327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-459n2\" (UniqueName: \"kubernetes.io/projected/ecc8553b-b5a2-482e-bb37-6281729826a2-kube-api-access-459n2\") pod \"console-operator-584dcb6b54-5568q\" (UID: \"ecc8553b-b5a2-482e-bb37-6281729826a2\") " pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.885816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.885167    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jcnwx\" (UniqueName: \"kubernetes.io/projected/ae3e4780-f446-480b-917b-d6d0aa02cfa8-kube-api-access-jcnwx\") pod \"service-ca-operator-5d6c75f84f-lg8vj\" (UID: \"ae3e4780-f446-480b-917b-d6d0aa02cfa8\") " pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.885816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.885553    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxfk7\" (UniqueName: \"kubernetes.io/projected/fab0b7f4-41fe-4684-b4fe-5497dfe59961-kube-api-access-hxfk7\") pod \"kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr\" (UID: \"fab0b7f4-41fe-4684-b4fe-5497dfe59961\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.892672 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.892487    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"
Jul 28 10:08:55.903645 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.903611    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"
Jul 28 10:08:55.922002 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.921981    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:08:55.964776 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.964748    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz"]
Jul 28 10:08:55.966454 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.966432    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m24lj\" (UniqueName: \"kubernetes.io/projected/492fda93-ef2b-4cb0-88c5-88e9598bc195-kube-api-access-m24lj\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:55.966545 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.966520    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:55.966663 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:55.966644    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 10:08:55.966740 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:55.966728    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls podName:492fda93-ef2b-4cb0-88c5-88e9598bc195 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:56.466693925 +0000 UTC m=+44.041886833 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls") pod "cluster-samples-operator-6748489656-5xf8z" (UID: "492fda93-ef2b-4cb0-88c5-88e9598bc195") : secret "samples-operator-tls" not found
Jul 28 10:08:55.967334 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:55.967313    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae99e605_6566_498c_87dc_a3c0c2bf3fbe.slice/crio-d368b2bdbdd7519901def1c1e61fafd225755aed42184cc334dc1c3dc04031b8 WatchSource:0}: Error finding container d368b2bdbdd7519901def1c1e61fafd225755aed42184cc334dc1c3dc04031b8: Status 404 returned error can't find the container with id d368b2bdbdd7519901def1c1e61fafd225755aed42184cc334dc1c3dc04031b8
Jul 28 10:08:55.977619 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:55.976600    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m24lj\" (UniqueName: \"kubernetes.io/projected/492fda93-ef2b-4cb0-88c5-88e9598bc195-kube-api-access-m24lj\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:56.061698 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:56.061552    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae3e4780_f446_480b_917b_d6d0aa02cfa8.slice/crio-a03a4caade0db77881eb54e1495b18a01ff05593253ff6f8dd6587a5ff2faf50 WatchSource:0}: Error finding container a03a4caade0db77881eb54e1495b18a01ff05593253ff6f8dd6587a5ff2faf50: Status 404 returned error can't find the container with id a03a4caade0db77881eb54e1495b18a01ff05593253ff6f8dd6587a5ff2faf50
Jul 28 10:08:56.062401 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:56.062361    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj"]
Jul 28 10:08:56.105582 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:56.105534    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr"]
Jul 28 10:08:56.108782 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:56.108753    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfab0b7f4_41fe_4684_b4fe_5497dfe59961.slice/crio-e84293b43ddde2ce71b3c296725f77b5003b1dff2c4b6bcf763bc61f0c199271 WatchSource:0}: Error finding container e84293b43ddde2ce71b3c296725f77b5003b1dff2c4b6bcf763bc61f0c199271: Status 404 returned error can't find the container with id e84293b43ddde2ce71b3c296725f77b5003b1dff2c4b6bcf763bc61f0c199271
Jul 28 10:08:56.122143 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:56.122118    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-584dcb6b54-5568q"]
Jul 28 10:08:56.125003 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:08:56.124977    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecc8553b_b5a2_482e_bb37_6281729826a2.slice/crio-4d09975f33bfbc072ab77ce3a99ac1435ba461a055e9d541c692b5487463bcb4 WatchSource:0}: Error finding container 4d09975f33bfbc072ab77ce3a99ac1435ba461a055e9d541c692b5487463bcb4: Status 404 returned error can't find the container with id 4d09975f33bfbc072ab77ce3a99ac1435ba461a055e9d541c692b5487463bcb4
Jul 28 10:08:56.168565 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:56.168532    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj" event={"ID":"ae3e4780-f446-480b-917b-d6d0aa02cfa8","Type":"ContainerStarted","Data":"a03a4caade0db77881eb54e1495b18a01ff05593253ff6f8dd6587a5ff2faf50"}
Jul 28 10:08:56.169621 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:56.169597    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5568q" event={"ID":"ecc8553b-b5a2-482e-bb37-6281729826a2","Type":"ContainerStarted","Data":"4d09975f33bfbc072ab77ce3a99ac1435ba461a055e9d541c692b5487463bcb4"}
Jul 28 10:08:56.170627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:56.170608    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr" event={"ID":"fab0b7f4-41fe-4684-b4fe-5497dfe59961","Type":"ContainerStarted","Data":"e84293b43ddde2ce71b3c296725f77b5003b1dff2c4b6bcf763bc61f0c199271"}
Jul 28 10:08:56.171687 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:56.171668    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz" event={"ID":"ae99e605-6566-498c-87dc-a3c0c2bf3fbe","Type":"ContainerStarted","Data":"d368b2bdbdd7519901def1c1e61fafd225755aed42184cc334dc1c3dc04031b8"}
Jul 28 10:08:56.469659 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:56.469620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:56.470078 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:56.469766    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 10:08:56.470078 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:56.469821    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls podName:492fda93-ef2b-4cb0-88c5-88e9598bc195 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:57.469806103 +0000 UTC m=+45.044999011 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls") pod "cluster-samples-operator-6748489656-5xf8z" (UID: "492fda93-ef2b-4cb0-88c5-88e9598bc195") : secret "samples-operator-tls" not found
Jul 28 10:08:57.477835 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:57.477737    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:57.478307 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:57.477964    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 10:08:57.478307 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:57.478038    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls podName:492fda93-ef2b-4cb0-88c5-88e9598bc195 nodeName:}" failed. No retries permitted until 2026-07-28 10:08:59.478019347 +0000 UTC m=+47.053212271 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls") pod "cluster-samples-operator-6748489656-5xf8z" (UID: "492fda93-ef2b-4cb0-88c5-88e9598bc195") : secret "samples-operator-tls" not found
Jul 28 10:08:57.982058 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:57.982017    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:57.982250 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:57.982102    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:08:57.982250 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:57.982162    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 28 10:08:57.982250 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:57.982236    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:09:05.982214611 +0000 UTC m=+53.557407533 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : configmap "kube-rbac-proxy" not found
Jul 28 10:08:57.982441 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:57.982286    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:57.982441 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:57.982334    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:09:05.982319025 +0000 UTC m=+53.557511934 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:08:59.494610 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:08:59.494568    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:08:59.495205 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:59.494755    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 10:08:59.495205 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:08:59.494832    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls podName:492fda93-ef2b-4cb0-88c5-88e9598bc195 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:03.494811171 +0000 UTC m=+51.070004091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls") pod "cluster-samples-operator-6748489656-5xf8z" (UID: "492fda93-ef2b-4cb0-88c5-88e9598bc195") : secret "samples-operator-tls" not found
Jul 28 10:09:01.187553 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.187521    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj" event={"ID":"ae3e4780-f446-480b-917b-d6d0aa02cfa8","Type":"ContainerStarted","Data":"bf38ba3eafcc1e3f03cd6201b5653d8666b525d9001dd59e4f07cf7f35185987"}
Jul 28 10:09:01.189565 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.189538    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/0.log"
Jul 28 10:09:01.189700 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.189588    2560 generic.go:356] "Generic (PLEG): container finished" podID="ecc8553b-b5a2-482e-bb37-6281729826a2" containerID="0fcc630d82ae39659cde218ce35a31a4941d8e586fb1d3730b47f3a434fb6aa3" exitCode=255
Jul 28 10:09:01.189880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.189859    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5568q" event={"ID":"ecc8553b-b5a2-482e-bb37-6281729826a2","Type":"ContainerDied","Data":"0fcc630d82ae39659cde218ce35a31a4941d8e586fb1d3730b47f3a434fb6aa3"}
Jul 28 10:09:01.189979 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.189957    2560 scope.go:117] "RemoveContainer" containerID="0fcc630d82ae39659cde218ce35a31a4941d8e586fb1d3730b47f3a434fb6aa3"
Jul 28 10:09:01.193998 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.193953    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr" event={"ID":"fab0b7f4-41fe-4684-b4fe-5497dfe59961","Type":"ContainerStarted","Data":"d1f1fc3b2caa48df35546298b756a82436541fc16a701e8ec958bccff9091bc2"}
Jul 28 10:09:01.198848 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.198825    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz" event={"ID":"ae99e605-6566-498c-87dc-a3c0c2bf3fbe","Type":"ContainerStarted","Data":"cfc5007ba74bc817bb3eea4c852f20e92c68ccaa82e2500624d5ff80f4763762"}
Jul 28 10:09:01.207104 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.207064    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj" podStartSLOduration=2.202109488 podStartE2EDuration="6.207051522s" totalImagesPullingTime="4.004942034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:01.206343006 +0000 UTC m=+48.781536021" watchObservedRunningTime="2026-07-28 10:09:01.207051522 +0000 UTC m=+48.782244451"
Jul 28 10:09:01.234520 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.234479    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-549c6d54cf-sg5vz" podStartSLOduration=2.13649272 podStartE2EDuration="6.23446296s" totalImagesPullingTime="4.09797024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:01.234161778 +0000 UTC m=+48.809354707" watchObservedRunningTime="2026-07-28 10:09:01.23446296 +0000 UTC m=+48.809655923"
Jul 28 10:09:01.278918 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:01.278871    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr" podStartSLOduration=2.321054142 podStartE2EDuration="6.278852674s" totalImagesPullingTime="3.957798532s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:01.277610187 +0000 UTC m=+48.852803118" watchObservedRunningTime="2026-07-28 10:09:01.278852674 +0000 UTC m=+48.854045605"
Jul 28 10:09:02.203688 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:02.203660    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:09:02.204131 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:02.204114    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/0.log"
Jul 28 10:09:02.204175 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:02.204149    2560 generic.go:356] "Generic (PLEG): container finished" podID="ecc8553b-b5a2-482e-bb37-6281729826a2" containerID="d060fcb1bd8a50ed22b5387ada1a5e22dd9645c1f0cf17b56a1bde4f90383243" exitCode=255
Jul 28 10:09:02.204205 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:02.204184    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5568q" event={"ID":"ecc8553b-b5a2-482e-bb37-6281729826a2","Type":"ContainerDied","Data":"d060fcb1bd8a50ed22b5387ada1a5e22dd9645c1f0cf17b56a1bde4f90383243"}
Jul 28 10:09:02.204260 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:02.204249    2560 scope.go:117] "RemoveContainer" containerID="0fcc630d82ae39659cde218ce35a31a4941d8e586fb1d3730b47f3a434fb6aa3"
Jul 28 10:09:02.204448 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:02.204426    2560 scope.go:117] "RemoveContainer" containerID="d060fcb1bd8a50ed22b5387ada1a5e22dd9645c1f0cf17b56a1bde4f90383243"
Jul 28 10:09:02.204641 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:02.204619    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-5568q_openshift-console-operator(ecc8553b-b5a2-482e-bb37-6281729826a2)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5568q" podUID="ecc8553b-b5a2-482e-bb37-6281729826a2"
Jul 28 10:09:02.521452 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:02.521364    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:09:02.521642 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:02.521528    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 28 10:09:02.521642 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:02.521603    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls podName:1416cd8c-7960-4cc9-8444-bc3ad41e74d2 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:18.521582664 +0000 UTC m=+66.096775588 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls") pod "dns-default-4gf6b" (UID: "1416cd8c-7960-4cc9-8444-bc3ad41e74d2") : secret "dns-default-metrics-tls" not found
Jul 28 10:09:03.208664 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:03.208635    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:09:03.209052 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:03.208968    2560 scope.go:117] "RemoveContainer" containerID="d060fcb1bd8a50ed22b5387ada1a5e22dd9645c1f0cf17b56a1bde4f90383243"
Jul 28 10:09:03.209128 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:03.209110    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-5568q_openshift-console-operator(ecc8553b-b5a2-482e-bb37-6281729826a2)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5568q" podUID="ecc8553b-b5a2-482e-bb37-6281729826a2"
Jul 28 10:09:03.528934 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:03.528855    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:09:03.529135 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:03.529005    2560 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 28 10:09:03.529135 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:03.529072    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls podName:492fda93-ef2b-4cb0-88c5-88e9598bc195 nodeName:}" failed. No retries permitted until 2026-07-28 10:09:11.529056079 +0000 UTC m=+59.104248985 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls") pod "cluster-samples-operator-6748489656-5xf8z" (UID: "492fda93-ef2b-4cb0-88c5-88e9598bc195") : secret "samples-operator-tls" not found
Jul 28 10:09:05.922955 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:05.922918    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:09:05.923426 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:05.922978    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:09:05.923426 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:05.923286    2560 scope.go:117] "RemoveContainer" containerID="d060fcb1bd8a50ed22b5387ada1a5e22dd9645c1f0cf17b56a1bde4f90383243"
Jul 28 10:09:05.923500 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:05.923442    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-584dcb6b54-5568q_openshift-console-operator(ecc8553b-b5a2-482e-bb37-6281729826a2)\"" pod="openshift-console-operator/console-operator-584dcb6b54-5568q" podUID="ecc8553b-b5a2-482e-bb37-6281729826a2"
Jul 28 10:09:06.048834 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:06.048799    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:09:06.048968 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:06.048853    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:09:06.048968 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:06.048940    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 28 10:09:06.049034 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:06.049004    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls podName:39fdcf2e-356b-49df-8e08-10a12febca9e nodeName:}" failed. No retries permitted until 2026-07-28 10:09:22.048987501 +0000 UTC m=+69.624180414 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls") pod "insights-runtime-extractor-gk59v" (UID: "39fdcf2e-356b-49df-8e08-10a12febca9e") : secret "insights-runtime-extractor-tls" not found
Jul 28 10:09:06.049299 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:06.049278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/39fdcf2e-356b-49df-8e08-10a12febca9e-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:09:11.588614 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:11.588560    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:09:11.591109 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:11.591072    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/492fda93-ef2b-4cb0-88c5-88e9598bc195-samples-operator-tls\") pod \"cluster-samples-operator-6748489656-5xf8z\" (UID: \"492fda93-ef2b-4cb0-88c5-88e9598bc195\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:09:11.886028 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:11.886001    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"
Jul 28 10:09:12.000527 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:12.000494    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z"]
Jul 28 10:09:12.231948 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:12.231881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z" event={"ID":"492fda93-ef2b-4cb0-88c5-88e9598bc195","Type":"ContainerStarted","Data":"ba95ed2f34e9018faf2c4dabb09aea64b46127d5c8c2d2076d32b7d27692b7bb"}
Jul 28 10:09:13.135790 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:13.135755    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-9cmbd"
Jul 28 10:09:14.239785 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:14.239749    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z" event={"ID":"492fda93-ef2b-4cb0-88c5-88e9598bc195","Type":"ContainerStarted","Data":"ea150bb67b96c8f0211211c9fddb20a5271f5fac5a3f82a0c6efcd393347438d"}
Jul 28 10:09:14.240121 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:14.239852    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z" event={"ID":"492fda93-ef2b-4cb0-88c5-88e9598bc195","Type":"ContainerStarted","Data":"7ab67641e8feaaa9474028fbb6d115d99a54c1130f274ca525226c9f86b5330a"}
Jul 28 10:09:14.257680 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:14.257530    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-6748489656-5xf8z" podStartSLOduration=17.177752709 podStartE2EDuration="19.257511315s" totalImagesPullingTime="2.079758606s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:14.256467256 +0000 UTC m=+61.831660200" watchObservedRunningTime="2026-07-28 10:09:14.257511315 +0000 UTC m=+61.832704245"
Jul 28 10:09:18.541616 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.541586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:09:18.543865 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.543843    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/1416cd8c-7960-4cc9-8444-bc3ad41e74d2-metrics-tls\") pod \"dns-default-4gf6b\" (UID: \"1416cd8c-7960-4cc9-8444-bc3ad41e74d2\") " pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:09:18.642140 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.642083    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:09:18.642238 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.642148    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:09:18.644554 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.644537    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 28 10:09:18.644632 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.644615    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 28 10:09:18.655317 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.655291    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ddff1dad-46bf-4209-97ff-8252a8347832-metrics-certs\") pod \"network-metrics-daemon-vxtwq\" (UID: \"ddff1dad-46bf-4209-97ff-8252a8347832\") " pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:09:18.655421 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.655406    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/74e6f51a-e169-4372-9ba9-48189d368e93-original-pull-secret\") pod \"global-pull-secret-syncer-t6srt\" (UID: \"74e6f51a-e169-4372-9ba9-48189d368e93\") " pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:09:18.743392 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.743358    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:09:18.745831 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.745816    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 28 10:09:18.756693 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.756677    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 28 10:09:18.762923 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.762906    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-kvgz4\""
Jul 28 10:09:18.767375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.767358    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-twrtv\" (UniqueName: \"kubernetes.io/projected/a667ef37-25e2-454b-97e2-aa2b32f5b28a-kube-api-access-twrtv\") pod \"network-check-target-lm85v\" (UID: \"a667ef37-25e2-454b-97e2-aa2b32f5b28a\") " pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:09:18.771156 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.771139    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:09:18.785555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.785538    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-t6srt"
Jul 28 10:09:18.793738 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.793676    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nwnjm\""
Jul 28 10:09:18.801402 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.801386    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:09:18.884104 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.884071    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tgfjj\""
Jul 28 10:09:18.891846 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.891768    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-vxtwq"
Jul 28 10:09:18.907541 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.907517    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-4gf6b"]
Jul 28 10:09:18.919630 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:09:18.919566    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1416cd8c_7960_4cc9_8444_bc3ad41e74d2.slice/crio-41b066e1471c82550d06edc18f5bd1efa1d566b3d00b90130f9ce997fc5bb53d WatchSource:0}: Error finding container 41b066e1471c82550d06edc18f5bd1efa1d566b3d00b90130f9ce997fc5bb53d: Status 404 returned error can't find the container with id 41b066e1471c82550d06edc18f5bd1efa1d566b3d00b90130f9ce997fc5bb53d
Jul 28 10:09:18.930847 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.930823    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-t6srt"]
Jul 28 10:09:18.933859 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:09:18.933835    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod74e6f51a_e169_4372_9ba9_48189d368e93.slice/crio-ea4fff939cbe5d841fc793e04ab016fccf955a746174aeadfd1d6c4db577f39c WatchSource:0}: Error finding container ea4fff939cbe5d841fc793e04ab016fccf955a746174aeadfd1d6c4db577f39c: Status 404 returned error can't find the container with id ea4fff939cbe5d841fc793e04ab016fccf955a746174aeadfd1d6c4db577f39c
Jul 28 10:09:18.947157 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.947122    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-lm85v"]
Jul 28 10:09:18.951073 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:09:18.951043    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda667ef37_25e2_454b_97e2_aa2b32f5b28a.slice/crio-ba021d081abe67c390163bbb335b4d6a4c4f16a3ecfff28fd585e251a4869fc5 WatchSource:0}: Error finding container ba021d081abe67c390163bbb335b4d6a4c4f16a3ecfff28fd585e251a4869fc5: Status 404 returned error can't find the container with id ba021d081abe67c390163bbb335b4d6a4c4f16a3ecfff28fd585e251a4869fc5
Jul 28 10:09:18.972492 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:18.972470    2560 scope.go:117] "RemoveContainer" containerID="d060fcb1bd8a50ed22b5387ada1a5e22dd9645c1f0cf17b56a1bde4f90383243"
Jul 28 10:09:19.027805 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.027783    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-vxtwq"]
Jul 28 10:09:19.031048 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:09:19.031026    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddff1dad_46bf_4209_97ff_8252a8347832.slice/crio-b9cf480ca4cf999df8cdcfcd6d77e6de5bbb0743c56d4b8319940ed5471eeee8 WatchSource:0}: Error finding container b9cf480ca4cf999df8cdcfcd6d77e6de5bbb0743c56d4b8319940ed5471eeee8: Status 404 returned error can't find the container with id b9cf480ca4cf999df8cdcfcd6d77e6de5bbb0743c56d4b8319940ed5471eeee8
Jul 28 10:09:19.259637 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.259609    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t6srt" event={"ID":"74e6f51a-e169-4372-9ba9-48189d368e93","Type":"ContainerStarted","Data":"ea4fff939cbe5d841fc793e04ab016fccf955a746174aeadfd1d6c4db577f39c"}
Jul 28 10:09:19.260651 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.260621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gf6b" event={"ID":"1416cd8c-7960-4cc9-8444-bc3ad41e74d2","Type":"ContainerStarted","Data":"41b066e1471c82550d06edc18f5bd1efa1d566b3d00b90130f9ce997fc5bb53d"}
Jul 28 10:09:19.261582 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.261564    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lm85v" event={"ID":"a667ef37-25e2-454b-97e2-aa2b32f5b28a","Type":"ContainerStarted","Data":"ba021d081abe67c390163bbb335b4d6a4c4f16a3ecfff28fd585e251a4869fc5"}
Jul 28 10:09:19.262610 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.262589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxtwq" event={"ID":"ddff1dad-46bf-4209-97ff-8252a8347832","Type":"ContainerStarted","Data":"b9cf480ca4cf999df8cdcfcd6d77e6de5bbb0743c56d4b8319940ed5471eeee8"}
Jul 28 10:09:19.264332 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.264313    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:09:19.264431 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.264354    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-584dcb6b54-5568q" event={"ID":"ecc8553b-b5a2-482e-bb37-6281729826a2","Type":"ContainerStarted","Data":"8d26ea904d31165f4c9ff6272ddf69f9dc014bb80b9b430809b6b22ccdde5e51"}
Jul 28 10:09:19.264621 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.264597    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:09:19.282457 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.282413    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-584dcb6b54-5568q" podStartSLOduration=20.33625244 podStartE2EDuration="24.282397906s" totalImagesPullingTime="3.946145466s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:19.281683005 +0000 UTC m=+66.856875936" watchObservedRunningTime="2026-07-28 10:09:19.282397906 +0000 UTC m=+66.857590834"
Jul 28 10:09:19.349581 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:19.349552    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-584dcb6b54-5568q"
Jul 28 10:09:21.277459 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:21.277377    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxtwq" event={"ID":"ddff1dad-46bf-4209-97ff-8252a8347832","Type":"ContainerStarted","Data":"564503bebf3a311351be9097b5ed8d35cc8f3c7d1b4033bb1fdc185b51f9da61"}
Jul 28 10:09:21.277459 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:21.277449    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-vxtwq" event={"ID":"ddff1dad-46bf-4209-97ff-8252a8347832","Type":"ContainerStarted","Data":"3ae91dace5ee67a3e7b5c08c76d29867be28db08bf86db3a56b97ef8b1fe054e"}
Jul 28 10:09:21.295013 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:21.294284    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-vxtwq" podStartSLOduration=67.054264521 podStartE2EDuration="1m8.294266261s" totalImagesPullingTime="1.24000174s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:21.293734178 +0000 UTC m=+68.868927102" watchObservedRunningTime="2026-07-28 10:09:21.294266261 +0000 UTC m=+68.869459194"
Jul 28 10:09:22.022843 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.022813    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"]
Jul 28 10:09:22.041789 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.041761    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"]
Jul 28 10:09:22.041951 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.041887    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.044819 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.044773    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 28 10:09:22.044952 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.044786    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 28 10:09:22.044952 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.044846    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 28 10:09:22.045272 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.045237    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 28 10:09:22.045272 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.045260    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 28 10:09:22.045441 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.045274    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 28 10:09:22.045681 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.045660    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 28 10:09:22.075218 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.075189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:09:22.077920 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.077900    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/39fdcf2e-356b-49df-8e08-10a12febca9e-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-gk59v\" (UID: \"39fdcf2e-356b-49df-8e08-10a12febca9e\") " pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:09:22.154787 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.154760    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5bfcf7795c-h4m2z"]
Jul 28 10:09:22.173014 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.172991    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.175604 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.175580    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-ca\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.175748 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.175622    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.175748 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.175657    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-hub\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.175748 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.175703    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.175985 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.175750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84vnd\" (UniqueName: \"kubernetes.io/projected/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-kube-api-access-84vnd\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.175985 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.175799    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.176106 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.176014    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 28 10:09:22.176196 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.176180    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 28 10:09:22.176375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.176014    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-bcfzv\""
Jul 28 10:09:22.176554 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.176536    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 28 10:09:22.177192 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.177137    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bfcf7795c-h4m2z"]
Jul 28 10:09:22.181348 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.181328    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 28 10:09:22.276810 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.276739    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-ca\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.276810 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.276778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.277030 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.276920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-bound-sa-token\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.277030 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.276968    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-hub\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.277030 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.277183 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84vnd\" (UniqueName: \"kubernetes.io/projected/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-kube-api-access-84vnd\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.277183 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277078    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8bd08be2-abd3-41e9-bbee-d9ab879633ce-installation-pull-secrets\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.277183 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8bd08be2-abd3-41e9-bbee-d9ab879633ce-ca-trust-extracted\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.277183 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277137    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8bd08be2-abd3-41e9-bbee-d9ab879633ce-registry-certificates\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.277183 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8bd08be2-abd3-41e9-bbee-d9ab879633ce-trusted-ca\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.277409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277228    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.277409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277273    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8bd08be2-abd3-41e9-bbee-d9ab879633ce-image-registry-private-configuration\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.277409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277309    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-registry-tls\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.277409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kr7n\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-kube-api-access-9kr7n\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.277936 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.277645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-additional-service-ca\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.279839 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.279795    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-ca\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.279839 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.279829    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-hub\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.280185 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.280162    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.280873 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.280855    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.286203 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.286178    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84vnd\" (UniqueName: \"kubernetes.io/projected/8c283804-c3dd-4d63-ae1b-c276c71ce8ea-kube-api-access-84vnd\") pod \"cluster-proxy-proxy-agent-56d748494b-ftfng\" (UID: \"8c283804-c3dd-4d63-ae1b-c276c71ce8ea\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.312026 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.312008    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-vsm7x\""
Jul 28 10:09:22.319971 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.319953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-gk59v"
Jul 28 10:09:22.366031 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.365999    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"
Jul 28 10:09:22.378765 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.378729    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8bd08be2-abd3-41e9-bbee-d9ab879633ce-image-registry-private-configuration\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.378865 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.378773    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-registry-tls\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.378865 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.378813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kr7n\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-kube-api-access-9kr7n\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.378865 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.378849    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-bound-sa-token\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.379011 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.378908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8bd08be2-abd3-41e9-bbee-d9ab879633ce-installation-pull-secrets\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.379011 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.378932    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8bd08be2-abd3-41e9-bbee-d9ab879633ce-ca-trust-extracted\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.379011 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.378959    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8bd08be2-abd3-41e9-bbee-d9ab879633ce-registry-certificates\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.379011 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.378990    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8bd08be2-abd3-41e9-bbee-d9ab879633ce-trusted-ca\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.380116 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.380034    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/8bd08be2-abd3-41e9-bbee-d9ab879633ce-trusted-ca\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.380218 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.380196    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/8bd08be2-abd3-41e9-bbee-d9ab879633ce-ca-trust-extracted\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.380781 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.380759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/8bd08be2-abd3-41e9-bbee-d9ab879633ce-registry-certificates\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.382691 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.382663    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/8bd08be2-abd3-41e9-bbee-d9ab879633ce-installation-pull-secrets\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.382691 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.382681    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-registry-tls\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.382957 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.382934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/8bd08be2-abd3-41e9-bbee-d9ab879633ce-image-registry-private-configuration\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.388385 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.388366    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-bound-sa-token\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.388784 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.388761    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kr7n\" (UniqueName: \"kubernetes.io/projected/8bd08be2-abd3-41e9-bbee-d9ab879633ce-kube-api-access-9kr7n\") pod \"image-registry-5bfcf7795c-h4m2z\" (UID: \"8bd08be2-abd3-41e9-bbee-d9ab879633ce\") " pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:22.485825 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:22.485782    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:24.379671 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:24.379614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng"]
Jul 28 10:09:24.386043 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:09:24.385349    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8c283804_c3dd_4d63_ae1b_c276c71ce8ea.slice/crio-7f00d47a69476dc01e4f40dd0cdcc8a4fc41e85a3385b9e8768f85993378cb00 WatchSource:0}: Error finding container 7f00d47a69476dc01e4f40dd0cdcc8a4fc41e85a3385b9e8768f85993378cb00: Status 404 returned error can't find the container with id 7f00d47a69476dc01e4f40dd0cdcc8a4fc41e85a3385b9e8768f85993378cb00
Jul 28 10:09:24.410699 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:24.410671    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5bfcf7795c-h4m2z"]
Jul 28 10:09:24.411965 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:24.411892    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-gk59v"]
Jul 28 10:09:24.415879 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:09:24.415855    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8bd08be2_abd3_41e9_bbee_d9ab879633ce.slice/crio-5af69973b72b74200e28e4ea2c342856ada2e6b0c4d36c724e1fd03293d45f33 WatchSource:0}: Error finding container 5af69973b72b74200e28e4ea2c342856ada2e6b0c4d36c724e1fd03293d45f33: Status 404 returned error can't find the container with id 5af69973b72b74200e28e4ea2c342856ada2e6b0c4d36c724e1fd03293d45f33
Jul 28 10:09:24.416990 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:09:24.416965    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39fdcf2e_356b_49df_8e08_10a12febca9e.slice/crio-d78692dd27c4c178ddbae9735f39fca66d4c919c56cb651e26f21db14eba9e3e WatchSource:0}: Error finding container d78692dd27c4c178ddbae9735f39fca66d4c919c56cb651e26f21db14eba9e3e: Status 404 returned error can't find the container with id d78692dd27c4c178ddbae9735f39fca66d4c919c56cb651e26f21db14eba9e3e
Jul 28 10:09:25.302124 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.302091    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gf6b" event={"ID":"1416cd8c-7960-4cc9-8444-bc3ad41e74d2","Type":"ContainerStarted","Data":"48870ef91b8591dd4d5fbf06fc9bb43be482079c9a53dc40d2acd463f593acdf"}
Jul 28 10:09:25.302297 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.302141    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-4gf6b" event={"ID":"1416cd8c-7960-4cc9-8444-bc3ad41e74d2","Type":"ContainerStarted","Data":"961736881af777604185952470e7fecc6a73ce92a3b320fb0613074901c87448"}
Jul 28 10:09:25.302297 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.302192    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:09:25.303676 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.303651    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z" event={"ID":"8bd08be2-abd3-41e9-bbee-d9ab879633ce","Type":"ContainerStarted","Data":"b039dd8cbd62bf287d825ab6c6099673ccad71b1ba1c106fd67780559a661d5a"}
Jul 28 10:09:25.303814 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.303688    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z" event={"ID":"8bd08be2-abd3-41e9-bbee-d9ab879633ce","Type":"ContainerStarted","Data":"5af69973b72b74200e28e4ea2c342856ada2e6b0c4d36c724e1fd03293d45f33"}
Jul 28 10:09:25.303814 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.303767    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:25.304745 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.304708    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk59v" event={"ID":"39fdcf2e-356b-49df-8e08-10a12febca9e","Type":"ContainerStarted","Data":"d78692dd27c4c178ddbae9735f39fca66d4c919c56cb651e26f21db14eba9e3e"}
Jul 28 10:09:25.306178 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.306159    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-lm85v" event={"ID":"a667ef37-25e2-454b-97e2-aa2b32f5b28a","Type":"ContainerStarted","Data":"c084117d2d2c5a63ff42d0f2fe9da4892f6b4017c9c5a4dbc3a99fdd72b70f12"}
Jul 28 10:09:25.306263 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.306203    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:09:25.307208 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.307189    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng" event={"ID":"8c283804-c3dd-4d63-ae1b-c276c71ce8ea","Type":"ContainerStarted","Data":"7f00d47a69476dc01e4f40dd0cdcc8a4fc41e85a3385b9e8768f85993378cb00"}
Jul 28 10:09:25.308282 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.308264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-t6srt" event={"ID":"74e6f51a-e169-4372-9ba9-48189d368e93","Type":"ContainerStarted","Data":"5ac6e4f4cea982913c27281979e27120637aedc3a68ecbb193ca80e5c0979711"}
Jul 28 10:09:25.321626 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.321586    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-4gf6b" podStartSLOduration=34.060445837 podStartE2EDuration="39.32157288s" totalImagesPullingTime="5.261127043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:25.320063529 +0000 UTC m=+72.895256458" watchObservedRunningTime="2026-07-28 10:09:25.32157288 +0000 UTC m=+72.896765809"
Jul 28 10:09:25.336121 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.336084    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-t6srt" podStartSLOduration=67.072846034 podStartE2EDuration="1m12.336072087s" totalImagesPullingTime="5.263226053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:25.334677623 +0000 UTC m=+72.909870552" watchObservedRunningTime="2026-07-28 10:09:25.336072087 +0000 UTC m=+72.911265007"
Jul 28 10:09:25.366880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.366848    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-lm85v" podStartSLOduration=67.133741281 podStartE2EDuration="1m12.36683664s" totalImagesPullingTime="5.233095359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:08:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:25.366347651 +0000 UTC m=+72.941540580" watchObservedRunningTime="2026-07-28 10:09:25.36683664 +0000 UTC m=+72.942029568"
Jul 28 10:09:25.367488 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:25.367457    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z" podStartSLOduration=3.367450162 podStartE2EDuration="3.367450162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:25.352215488 +0000 UTC m=+72.927408418" watchObservedRunningTime="2026-07-28 10:09:25.367450162 +0000 UTC m=+72.942643091"
Jul 28 10:09:31.499479 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.499445    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-2tq4s"]
Jul 28 10:09:31.546837 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.546333    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.548835 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.548789    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 28 10:09:31.548835 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.548811    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 28 10:09:31.549073 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.548809    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 28 10:09:31.549204 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.549184    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 28 10:09:31.549301 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.549243    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 28 10:09:31.549672 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.549655    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-nvjpc\""
Jul 28 10:09:31.549785 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.549655    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 28 10:09:31.662472 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662439    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-wtmp\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.662632 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-metrics-client-ca\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.662632 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662566    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk5kw\" (UniqueName: \"kubernetes.io/projected/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-kube-api-access-hk5kw\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.662632 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662616    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-sys\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.662820 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662644    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-textfile\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.662820 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662760    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.662912 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662819    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.662912 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662850    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-tls\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.662912 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.662897    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-root\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763469 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-sys\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763654 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763482    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-textfile\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763654 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763654 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763560    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-sys\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763654 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763564    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763654 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763635    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-tls\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763947 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763667    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-root\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763947 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763705    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-wtmp\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763947 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763759    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-metrics-client-ca\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763947 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763792    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk5kw\" (UniqueName: \"kubernetes.io/projected/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-kube-api-access-hk5kw\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763947 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763798    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-root\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.763947 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:31.763812    2560 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 28 10:09:31.763947 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.763934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-textfile\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.764269 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:09:31.764022    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-tls podName:1586e95a-6b9f-4504-8094-9a6f47b0a5fc nodeName:}" failed. No retries permitted until 2026-07-28 10:09:32.263998296 +0000 UTC m=+79.839191218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-tls") pod "node-exporter-2tq4s" (UID: "1586e95a-6b9f-4504-8094-9a6f47b0a5fc") : secret "node-exporter-tls" not found
Jul 28 10:09:31.764269 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.764032    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-wtmp\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.764269 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.764251    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-accelerators-collector-config\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.764399 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.764334    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-metrics-client-ca\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.766160 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.766135    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:31.773182 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:31.773163    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk5kw\" (UniqueName: \"kubernetes.io/projected/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-kube-api-access-hk5kw\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:32.268424 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:32.268326    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-tls\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:32.270545 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:32.270517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/1586e95a-6b9f-4504-8094-9a6f47b0a5fc-node-exporter-tls\") pod \"node-exporter-2tq4s\" (UID: \"1586e95a-6b9f-4504-8094-9a6f47b0a5fc\") " pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:32.339366 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:32.339324    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk59v" event={"ID":"39fdcf2e-356b-49df-8e08-10a12febca9e","Type":"ContainerStarted","Data":"2d9421dbb9a7ac18ca33944e35fd932027d1e2c0f469fc6ace25ff6dd31dc418"}
Jul 28 10:09:32.341335 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:32.341310    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng" event={"ID":"8c283804-c3dd-4d63-ae1b-c276c71ce8ea","Type":"ContainerStarted","Data":"749c8c8fb20b74bb2eeb3bcf363d20793140de5fd9d965f908073d47a1c562f3"}
Jul 28 10:09:32.341439 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:32.341340    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng" event={"ID":"8c283804-c3dd-4d63-ae1b-c276c71ce8ea","Type":"ContainerStarted","Data":"55fdab135cadcf844bb8cbc647778f2112766bb9c27b7d2c16018be6d46444af"}
Jul 28 10:09:32.341439 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:32.341353    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng" event={"ID":"8c283804-c3dd-4d63-ae1b-c276c71ce8ea","Type":"ContainerStarted","Data":"67c1312de25e8c91f84469835924a3cf02d4eb4944be91e7b6de0a6114d2c45b"}
Jul 28 10:09:32.365024 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:32.364987    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-56d748494b-ftfng" podStartSLOduration=3.81411698 podStartE2EDuration="11.364970766s" totalImagesPullingTime="7.550853786s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:09:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:32.36380675 +0000 UTC m=+79.938999680" watchObservedRunningTime="2026-07-28 10:09:32.364970766 +0000 UTC m=+79.940163694"
Jul 28 10:09:32.458183 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:32.458158    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-2tq4s"
Jul 28 10:09:32.467298 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:09:32.467267    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1586e95a_6b9f_4504_8094_9a6f47b0a5fc.slice/crio-80ccc13e78b96e944e82c68158ad32661df05be179ab320a59e790d654c13ffd WatchSource:0}: Error finding container 80ccc13e78b96e944e82c68158ad32661df05be179ab320a59e790d654c13ffd: Status 404 returned error can't find the container with id 80ccc13e78b96e944e82c68158ad32661df05be179ab320a59e790d654c13ffd
Jul 28 10:09:33.348551 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:33.348510    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2tq4s" event={"ID":"1586e95a-6b9f-4504-8094-9a6f47b0a5fc","Type":"ContainerStarted","Data":"80ccc13e78b96e944e82c68158ad32661df05be179ab320a59e790d654c13ffd"}
Jul 28 10:09:34.354913 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:34.354777    2560 generic.go:356] "Generic (PLEG): container finished" podID="1586e95a-6b9f-4504-8094-9a6f47b0a5fc" containerID="c887709ac56742faee43088a724f14b571afd91f54bfbe87aecf4725aac5806b" exitCode=0
Jul 28 10:09:34.354913 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:34.354836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2tq4s" event={"ID":"1586e95a-6b9f-4504-8094-9a6f47b0a5fc","Type":"ContainerDied","Data":"c887709ac56742faee43088a724f14b571afd91f54bfbe87aecf4725aac5806b"}
Jul 28 10:09:35.318360 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:35.318327    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-4gf6b"
Jul 28 10:09:36.364394 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:36.364356    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk59v" event={"ID":"39fdcf2e-356b-49df-8e08-10a12febca9e","Type":"ContainerStarted","Data":"1ad694d0a8bcef6fa48d462dd0f481eb2650b70e1858b72641c956d075b5d957"}
Jul 28 10:09:36.366676 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:36.366648    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2tq4s" event={"ID":"1586e95a-6b9f-4504-8094-9a6f47b0a5fc","Type":"ContainerStarted","Data":"da804ba9d6d015defdc886e1d3069b1df6d3fbda072aaa0ee7de02a9f5e457ef"}
Jul 28 10:09:36.366813 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:36.366696    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-2tq4s" event={"ID":"1586e95a-6b9f-4504-8094-9a6f47b0a5fc","Type":"ContainerStarted","Data":"d0387f9ff6e17f0e07848701af247467b742a30ccd67a90fbab3d6b790e57553"}
Jul 28 10:09:36.388166 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:36.388118    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-2tq4s" podStartSLOduration=4.337533049 podStartE2EDuration="5.388098656s" totalImagesPullingTime="975.557741ms" totalInitContainerRuntime="75.007866ms" isStatefulPod=true podCreationTimestamp="2026-07-28 10:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:36.386788055 +0000 UTC m=+83.961980982" watchObservedRunningTime="2026-07-28 10:09:36.388098656 +0000 UTC m=+83.963291585"
Jul 28 10:09:39.381240 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:39.381206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-gk59v" event={"ID":"39fdcf2e-356b-49df-8e08-10a12febca9e","Type":"ContainerStarted","Data":"143673dc5dd22929de1581b112845d3bd4c07dfc360c5c8046866d2dea45081c"}
Jul 28 10:09:39.399089 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:39.399037    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-gk59v" podStartSLOduration=34.813789621 podStartE2EDuration="49.399020701s" totalImagesPullingTime="14.58523108s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:08:50 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:09:39.397960877 +0000 UTC m=+86.973153806" watchObservedRunningTime="2026-07-28 10:09:39.399020701 +0000 UTC m=+86.974213630"
Jul 28 10:09:46.319915 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:46.319883    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5bfcf7795c-h4m2z"
Jul 28 10:09:56.319067 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:09:56.319034    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-lm85v"
Jul 28 10:10:16.507045 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:10:16.507012    2560 generic.go:356] "Generic (PLEG): container finished" podID="ae3e4780-f446-480b-917b-d6d0aa02cfa8" containerID="bf38ba3eafcc1e3f03cd6201b5653d8666b525d9001dd59e4f07cf7f35185987" exitCode=0
Jul 28 10:10:16.507473 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:10:16.507086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj" event={"ID":"ae3e4780-f446-480b-917b-d6d0aa02cfa8","Type":"ContainerDied","Data":"bf38ba3eafcc1e3f03cd6201b5653d8666b525d9001dd59e4f07cf7f35185987"}
Jul 28 10:10:16.507473 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:10:16.507382    2560 scope.go:117] "RemoveContainer" containerID="bf38ba3eafcc1e3f03cd6201b5653d8666b525d9001dd59e4f07cf7f35185987"
Jul 28 10:10:17.512335 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:10:17.512297    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-5d6c75f84f-lg8vj" event={"ID":"ae3e4780-f446-480b-917b-d6d0aa02cfa8","Type":"ContainerStarted","Data":"4b3b8f4c3511b2336a11320b2ee967a79b27819d8b924b34456faaaed04c7430"}
Jul 28 10:10:31.562289 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:10:31.562253    2560 generic.go:356] "Generic (PLEG): container finished" podID="fab0b7f4-41fe-4684-b4fe-5497dfe59961" containerID="d1f1fc3b2caa48df35546298b756a82436541fc16a701e8ec958bccff9091bc2" exitCode=0
Jul 28 10:10:31.562289 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:10:31.562296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr" event={"ID":"fab0b7f4-41fe-4684-b4fe-5497dfe59961","Type":"ContainerDied","Data":"d1f1fc3b2caa48df35546298b756a82436541fc16a701e8ec958bccff9091bc2"}
Jul 28 10:10:31.562746 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:10:31.562561    2560 scope.go:117] "RemoveContainer" containerID="d1f1fc3b2caa48df35546298b756a82436541fc16a701e8ec958bccff9091bc2"
Jul 28 10:10:32.567987 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:10:32.567955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr" event={"ID":"fab0b7f4-41fe-4684-b4fe-5497dfe59961","Type":"ContainerStarted","Data":"257593df3665864ba33f25e4b5a056dc982fe9cdf5f6c68172afa48915395d13"}
Jul 28 10:13:12.818904 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:13:12.818877    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:13:12.824650 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:13:12.824627    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:13:12.829165 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:13:12.829146    2560 kubelet.go:1648] "Image garbage collection succeeded"
Jul 28 10:16:19.543011 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.542965    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6968749498-4l9px"]
Jul 28 10:16:19.545086 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.545063    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:19.547375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.547350    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 28 10:16:19.548199 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.548179    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 28 10:16:19.548199 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.548196    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-wtp65\""
Jul 28 10:16:19.548362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.548258    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 28 10:16:19.556695 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.556673    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-4l9px"]
Jul 28 10:16:19.577620 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.577593    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-9qjn5"]
Jul 28 10:16:19.583440 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.583421    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:19.585370 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.585351    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-xcbwm\""
Jul 28 10:16:19.585608 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.585594    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 28 10:16:19.589961 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.589941    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-9qjn5"]
Jul 28 10:16:19.706090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.706044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z48d7\" (UniqueName: \"kubernetes.io/projected/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-kube-api-access-z48d7\") pod \"kserve-controller-manager-6968749498-4l9px\" (UID: \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\") " pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:19.706090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.706095    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ddfab1b-2429-44c8-b92f-de0de7169e82-data\") pod \"seaweedfs-64568bcd49-9qjn5\" (UID: \"0ddfab1b-2429-44c8-b92f-de0de7169e82\") " pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:19.706365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.706138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-cert\") pod \"kserve-controller-manager-6968749498-4l9px\" (UID: \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\") " pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:19.706365 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.706186    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zx7r2\" (UniqueName: \"kubernetes.io/projected/0ddfab1b-2429-44c8-b92f-de0de7169e82-kube-api-access-zx7r2\") pod \"seaweedfs-64568bcd49-9qjn5\" (UID: \"0ddfab1b-2429-44c8-b92f-de0de7169e82\") " pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:19.807151 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.807061    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ddfab1b-2429-44c8-b92f-de0de7169e82-data\") pod \"seaweedfs-64568bcd49-9qjn5\" (UID: \"0ddfab1b-2429-44c8-b92f-de0de7169e82\") " pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:19.807151 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.807098    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-cert\") pod \"kserve-controller-manager-6968749498-4l9px\" (UID: \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\") " pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:19.807151 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.807127    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zx7r2\" (UniqueName: \"kubernetes.io/projected/0ddfab1b-2429-44c8-b92f-de0de7169e82-kube-api-access-zx7r2\") pod \"seaweedfs-64568bcd49-9qjn5\" (UID: \"0ddfab1b-2429-44c8-b92f-de0de7169e82\") " pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:19.807393 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.807173    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z48d7\" (UniqueName: \"kubernetes.io/projected/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-kube-api-access-z48d7\") pod \"kserve-controller-manager-6968749498-4l9px\" (UID: \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\") " pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:19.807450 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.807429    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/0ddfab1b-2429-44c8-b92f-de0de7169e82-data\") pod \"seaweedfs-64568bcd49-9qjn5\" (UID: \"0ddfab1b-2429-44c8-b92f-de0de7169e82\") " pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:19.809502 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.809473    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-cert\") pod \"kserve-controller-manager-6968749498-4l9px\" (UID: \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\") " pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:19.817039 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.817018    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z48d7\" (UniqueName: \"kubernetes.io/projected/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-kube-api-access-z48d7\") pod \"kserve-controller-manager-6968749498-4l9px\" (UID: \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\") " pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:19.817134 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.817104    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zx7r2\" (UniqueName: \"kubernetes.io/projected/0ddfab1b-2429-44c8-b92f-de0de7169e82-kube-api-access-zx7r2\") pod \"seaweedfs-64568bcd49-9qjn5\" (UID: \"0ddfab1b-2429-44c8-b92f-de0de7169e82\") " pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:19.854934 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.854908    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:19.892786 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.892756    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:19.992460 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:16:19.992423    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c4b8f0e_de3a_453f_9922_6067ddcbbf3a.slice/crio-b3bb2aafd637cc069e957c881ddd33fdb5d69af19bc87cf1d0e35cca10519575 WatchSource:0}: Error finding container b3bb2aafd637cc069e957c881ddd33fdb5d69af19bc87cf1d0e35cca10519575: Status 404 returned error can't find the container with id b3bb2aafd637cc069e957c881ddd33fdb5d69af19bc87cf1d0e35cca10519575
Jul 28 10:16:19.992825 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.992799    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-4l9px"]
Jul 28 10:16:19.994039 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:19.994020    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:16:20.037428 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:20.037398    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-9qjn5"]
Jul 28 10:16:20.040239 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:16:20.040203    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ddfab1b_2429_44c8_b92f_de0de7169e82.slice/crio-fa27784c96b7d84eb740cbd76d41bea8c2b39ab9d199a900f722a82191d528f2 WatchSource:0}: Error finding container fa27784c96b7d84eb740cbd76d41bea8c2b39ab9d199a900f722a82191d528f2: Status 404 returned error can't find the container with id fa27784c96b7d84eb740cbd76d41bea8c2b39ab9d199a900f722a82191d528f2
Jul 28 10:16:20.725798 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:20.725730    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-9qjn5" event={"ID":"0ddfab1b-2429-44c8-b92f-de0de7169e82","Type":"ContainerStarted","Data":"fa27784c96b7d84eb740cbd76d41bea8c2b39ab9d199a900f722a82191d528f2"}
Jul 28 10:16:20.727147 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:20.727119    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-4l9px" event={"ID":"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a","Type":"ContainerStarted","Data":"b3bb2aafd637cc069e957c881ddd33fdb5d69af19bc87cf1d0e35cca10519575"}
Jul 28 10:16:23.740932 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:23.740895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-9qjn5" event={"ID":"0ddfab1b-2429-44c8-b92f-de0de7169e82","Type":"ContainerStarted","Data":"d0892f6da0d14908e0e05e8a7308455a48fb191203969321dc4b8ae0151fc7b0"}
Jul 28 10:16:23.741347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:23.741021    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:23.742399 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:23.742372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-4l9px" event={"ID":"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a","Type":"ContainerStarted","Data":"45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9"}
Jul 28 10:16:23.742530 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:23.742440    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:16:23.757291 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:23.757250    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-9qjn5" podStartSLOduration=1.271980878 podStartE2EDuration="4.757236412s" totalImagesPullingTime="3.485255534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:16:23.755941151 +0000 UTC m=+491.331134079" watchObservedRunningTime="2026-07-28 10:16:23.757236412 +0000 UTC m=+491.332429409"
Jul 28 10:16:23.771336 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:23.771291    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6968749498-4l9px" podStartSLOduration=1.305307915 podStartE2EDuration="4.771279441s" totalImagesPullingTime="3.465971526s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:16:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:16:23.771020151 +0000 UTC m=+491.346213078" watchObservedRunningTime="2026-07-28 10:16:23.771279441 +0000 UTC m=+491.346472370"
Jul 28 10:16:29.748573 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:29.748544    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-9qjn5"
Jul 28 10:16:54.751258 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:16:54.751232    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:17:01.430073 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.430038    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6968749498-4l9px"]
Jul 28 10:17:01.430623 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.430239    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-6968749498-4l9px" podUID="1c4b8f0e-de3a-453f-9922-6067ddcbbf3a" containerName="manager" containerID="cri-o://45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9" gracePeriod=10
Jul 28 10:17:01.463163 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.463138    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-6968749498-vpx4g"]
Jul 28 10:17:01.465139 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.465123    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:01.479163 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.479141    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-vpx4g"]
Jul 28 10:17:01.508308 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.508283    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c22rn\" (UniqueName: \"kubernetes.io/projected/f88840aa-df6f-45db-b54f-253a375fa2fe-kube-api-access-c22rn\") pod \"kserve-controller-manager-6968749498-vpx4g\" (UID: \"f88840aa-df6f-45db-b54f-253a375fa2fe\") " pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:01.508419 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.508318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88840aa-df6f-45db-b54f-253a375fa2fe-cert\") pod \"kserve-controller-manager-6968749498-vpx4g\" (UID: \"f88840aa-df6f-45db-b54f-253a375fa2fe\") " pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:01.609285 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.609257    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c22rn\" (UniqueName: \"kubernetes.io/projected/f88840aa-df6f-45db-b54f-253a375fa2fe-kube-api-access-c22rn\") pod \"kserve-controller-manager-6968749498-vpx4g\" (UID: \"f88840aa-df6f-45db-b54f-253a375fa2fe\") " pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:01.609420 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.609311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88840aa-df6f-45db-b54f-253a375fa2fe-cert\") pod \"kserve-controller-manager-6968749498-vpx4g\" (UID: \"f88840aa-df6f-45db-b54f-253a375fa2fe\") " pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:01.612042 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.612011    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f88840aa-df6f-45db-b54f-253a375fa2fe-cert\") pod \"kserve-controller-manager-6968749498-vpx4g\" (UID: \"f88840aa-df6f-45db-b54f-253a375fa2fe\") " pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:01.619028 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.619004    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c22rn\" (UniqueName: \"kubernetes.io/projected/f88840aa-df6f-45db-b54f-253a375fa2fe-kube-api-access-c22rn\") pod \"kserve-controller-manager-6968749498-vpx4g\" (UID: \"f88840aa-df6f-45db-b54f-253a375fa2fe\") " pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:01.668902 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.668880    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:17:01.710710 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.710645    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-cert\") pod \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\" (UID: \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\") "
Jul 28 10:17:01.710850 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.710745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z48d7\" (UniqueName: \"kubernetes.io/projected/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-kube-api-access-z48d7\") pod \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\" (UID: \"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a\") "
Jul 28 10:17:01.712805 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.712769    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-cert" (OuterVolumeSpecName: "cert") pod "1c4b8f0e-de3a-453f-9922-6067ddcbbf3a" (UID: "1c4b8f0e-de3a-453f-9922-6067ddcbbf3a"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:17:01.712921 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.712845    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-kube-api-access-z48d7" (OuterVolumeSpecName: "kube-api-access-z48d7") pod "1c4b8f0e-de3a-453f-9922-6067ddcbbf3a" (UID: "1c4b8f0e-de3a-453f-9922-6067ddcbbf3a"). InnerVolumeSpecName "kube-api-access-z48d7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:17:01.811630 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.811579    2560 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-cert\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:17:01.811630 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.811623    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z48d7\" (UniqueName: \"kubernetes.io/projected/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a-kube-api-access-z48d7\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:17:01.822701 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.822674    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:01.877081 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.877051    2560 generic.go:356] "Generic (PLEG): container finished" podID="1c4b8f0e-de3a-453f-9922-6067ddcbbf3a" containerID="45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9" exitCode=0
Jul 28 10:17:01.877238 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.877110    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-6968749498-4l9px"
Jul 28 10:17:01.877238 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.877163    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-4l9px" event={"ID":"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a","Type":"ContainerDied","Data":"45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9"}
Jul 28 10:17:01.877238 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.877211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-4l9px" event={"ID":"1c4b8f0e-de3a-453f-9922-6067ddcbbf3a","Type":"ContainerDied","Data":"b3bb2aafd637cc069e957c881ddd33fdb5d69af19bc87cf1d0e35cca10519575"}
Jul 28 10:17:01.877410 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.877241    2560 scope.go:117] "RemoveContainer" containerID="45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9"
Jul 28 10:17:01.888389 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.888373    2560 scope.go:117] "RemoveContainer" containerID="45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9"
Jul 28 10:17:01.888778 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:17:01.888749    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9\": container with ID starting with 45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9 not found: ID does not exist" containerID="45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9"
Jul 28 10:17:01.888882 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.888791    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9"} err="failed to get container status \"45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9\": rpc error: code = NotFound desc = could not find container \"45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9\": container with ID starting with 45fcfd6e4908d4e39f3680015a532823b2f13a6976bb6b92359a11a977be22d9 not found: ID does not exist"
Jul 28 10:17:01.904385 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.904358    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-6968749498-4l9px"]
Jul 28 10:17:01.907353 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.907327    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-6968749498-4l9px"]
Jul 28 10:17:01.950205 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:01.950181    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-6968749498-vpx4g"]
Jul 28 10:17:01.952564 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:17:01.952540    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf88840aa_df6f_45db_b54f_253a375fa2fe.slice/crio-7c2378d92f670063c07b5707e74e8c4dd43f527480fd3f1161c2ca75227dff04 WatchSource:0}: Error finding container 7c2378d92f670063c07b5707e74e8c4dd43f527480fd3f1161c2ca75227dff04: Status 404 returned error can't find the container with id 7c2378d92f670063c07b5707e74e8c4dd43f527480fd3f1161c2ca75227dff04
Jul 28 10:17:02.882173 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:02.882139    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-vpx4g" event={"ID":"f88840aa-df6f-45db-b54f-253a375fa2fe","Type":"ContainerStarted","Data":"829c082fcd648599980ff7d072eb10b3709975a7247c0067b32e5dcd03a19466"}
Jul 28 10:17:02.882173 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:02.882194    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-6968749498-vpx4g" event={"ID":"f88840aa-df6f-45db-b54f-253a375fa2fe","Type":"ContainerStarted","Data":"7c2378d92f670063c07b5707e74e8c4dd43f527480fd3f1161c2ca75227dff04"}
Jul 28 10:17:02.882661 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:02.882215    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:02.898631 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:02.898589    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-6968749498-vpx4g" podStartSLOduration=1.55431594 podStartE2EDuration="1.898574735s" totalImagesPullingTime="344.258795ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:02.898126769 +0000 UTC m=+530.473319700" watchObservedRunningTime="2026-07-28 10:17:02.898574735 +0000 UTC m=+530.473767665"
Jul 28 10:17:02.977709 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:02.977677    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1c4b8f0e-de3a-453f-9922-6067ddcbbf3a" path="/var/lib/kubelet/pods/1c4b8f0e-de3a-453f-9922-6067ddcbbf3a/volumes"
Jul 28 10:17:33.892959 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:33.892928    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-6968749498-vpx4g"
Jul 28 10:17:34.943939 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:34.943911    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-xvh7j"]
Jul 28 10:17:34.944321 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:34.944179    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1c4b8f0e-de3a-453f-9922-6067ddcbbf3a" containerName="manager"
Jul 28 10:17:34.944321 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:34.944190    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1c4b8f0e-de3a-453f-9922-6067ddcbbf3a" containerName="manager"
Jul 28 10:17:34.944321 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:34.944250    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1c4b8f0e-de3a-453f-9922-6067ddcbbf3a" containerName="manager"
Jul 28 10:17:34.946945 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:34.946928    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:34.949153 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:34.949127    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-webhook-cert\""
Jul 28 10:17:34.949376 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:34.949359    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"odh-model-controller-dockercfg-6zdmn\""
Jul 28 10:17:34.958299 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:34.958277    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-xvh7j"]
Jul 28 10:17:35.060115 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.060082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6xwl\" (UniqueName: \"kubernetes.io/projected/2506c208-d1a3-4c69-a08b-f82ed2bb6d79-kube-api-access-p6xwl\") pod \"odh-model-controller-75d9cdbddf-xvh7j\" (UID: \"2506c208-d1a3-4c69-a08b-f82ed2bb6d79\") " pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:35.060275 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.060126    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2506c208-d1a3-4c69-a08b-f82ed2bb6d79-cert\") pod \"odh-model-controller-75d9cdbddf-xvh7j\" (UID: \"2506c208-d1a3-4c69-a08b-f82ed2bb6d79\") " pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:35.160913 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.160879    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6xwl\" (UniqueName: \"kubernetes.io/projected/2506c208-d1a3-4c69-a08b-f82ed2bb6d79-kube-api-access-p6xwl\") pod \"odh-model-controller-75d9cdbddf-xvh7j\" (UID: \"2506c208-d1a3-4c69-a08b-f82ed2bb6d79\") " pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:35.161074 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.160923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2506c208-d1a3-4c69-a08b-f82ed2bb6d79-cert\") pod \"odh-model-controller-75d9cdbddf-xvh7j\" (UID: \"2506c208-d1a3-4c69-a08b-f82ed2bb6d79\") " pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:35.163293 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.163264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/2506c208-d1a3-4c69-a08b-f82ed2bb6d79-cert\") pod \"odh-model-controller-75d9cdbddf-xvh7j\" (UID: \"2506c208-d1a3-4c69-a08b-f82ed2bb6d79\") " pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:35.171067 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.171042    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6xwl\" (UniqueName: \"kubernetes.io/projected/2506c208-d1a3-4c69-a08b-f82ed2bb6d79-kube-api-access-p6xwl\") pod \"odh-model-controller-75d9cdbddf-xvh7j\" (UID: \"2506c208-d1a3-4c69-a08b-f82ed2bb6d79\") " pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:35.256871 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.256809    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:35.380414 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.380381    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/odh-model-controller-75d9cdbddf-xvh7j"]
Jul 28 10:17:35.383187 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:17:35.383161    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2506c208_d1a3_4c69_a08b_f82ed2bb6d79.slice/crio-89767472ddfd3f6f0b0c50d7c4d3ff6d4ba986a7fd68ff161622634f33e27040 WatchSource:0}: Error finding container 89767472ddfd3f6f0b0c50d7c4d3ff6d4ba986a7fd68ff161622634f33e27040: Status 404 returned error can't find the container with id 89767472ddfd3f6f0b0c50d7c4d3ff6d4ba986a7fd68ff161622634f33e27040
Jul 28 10:17:35.995625 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:35.995576    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-xvh7j" event={"ID":"2506c208-d1a3-4c69-a08b-f82ed2bb6d79","Type":"ContainerStarted","Data":"89767472ddfd3f6f0b0c50d7c4d3ff6d4ba986a7fd68ff161622634f33e27040"}
Jul 28 10:17:38.005596 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:38.005551    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/odh-model-controller-75d9cdbddf-xvh7j" event={"ID":"2506c208-d1a3-4c69-a08b-f82ed2bb6d79","Type":"ContainerStarted","Data":"35c9b0a3353de84cafb6f440a7f937992c2fe17ce08b99f6c84ec9b9264666b2"}
Jul 28 10:17:38.006046 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:38.005688    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:17:38.029504 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:38.029460    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/odh-model-controller-75d9cdbddf-xvh7j" podStartSLOduration=1.556481349 podStartE2EDuration="4.029442733s" totalImagesPullingTime="2.472961384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:17:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:17:38.02843087 +0000 UTC m=+565.603623800" watchObservedRunningTime="2026-07-28 10:17:38.029442733 +0000 UTC m=+565.604635663"
Jul 28 10:17:49.011681 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:17:49.011654    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/odh-model-controller-75d9cdbddf-xvh7j"
Jul 28 10:18:12.842982 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:18:12.842956    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:18:12.846047 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:18:12.846024    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:19:20.468295 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.468258    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"]
Jul 28 10:19:20.471661 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.471643    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.474463 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.474446    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 28 10:19:20.474799 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.474780    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 28 10:19:20.474866 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.474799    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-717c3-kube-rbac-proxy-sar-config\""
Jul 28 10:19:20.475331 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.475309    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-717c3-predictor-serving-cert\""
Jul 28 10:19:20.475406 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.475347    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-lvvzp\""
Jul 28 10:19:20.488999 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.488979    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"]
Jul 28 10:19:20.510609 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.510583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvn7s\" (UniqueName: \"kubernetes.io/projected/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-api-access-wvn7s\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.510702 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.510615    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.510702 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.510682    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.611821 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.611789    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvn7s\" (UniqueName: \"kubernetes.io/projected/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-api-access-wvn7s\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.612016 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.611834    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.612016 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.611908    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.612151 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:19:20.612026    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-717c3-predictor-serving-cert: secret "success-200-isvc-717c3-predictor-serving-cert" not found
Jul 28 10:19:20.612151 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:19:20.612120    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls podName:ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d nodeName:}" failed. No retries permitted until 2026-07-28 10:19:21.112099305 +0000 UTC m=+668.687292212 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls") pod "success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" (UID: "ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d") : secret "success-200-isvc-717c3-predictor-serving-cert" not found
Jul 28 10:19:20.612514 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.612494    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.623763 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.623736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvn7s\" (UniqueName: \"kubernetes.io/projected/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-api-access-wvn7s\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:20.868519 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.868448    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"]
Jul 28 10:19:20.871665 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.871647    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:20.874619 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.874597    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-717c3-predictor-serving-cert\""
Jul 28 10:19:20.874870 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.874853    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-717c3-kube-rbac-proxy-sar-config\""
Jul 28 10:19:20.883205 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.883184    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"]
Jul 28 10:19:20.914009 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.913989    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:20.914122 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.914034    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-proxy-tls\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:20.914122 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:20.914066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqvs4\" (UniqueName: \"kubernetes.io/projected/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-api-access-rqvs4\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:21.015069 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.015039    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqvs4\" (UniqueName: \"kubernetes.io/projected/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-api-access-rqvs4\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:21.015230 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.015208    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:21.015307 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.015291    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-proxy-tls\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:21.015897 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.015879    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:21.017681 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.017655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-proxy-tls\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:21.023903 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.023880    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqvs4\" (UniqueName: \"kubernetes.io/projected/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-api-access-rqvs4\") pod \"error-404-isvc-717c3-predictor-67f5fb9455-8qnmw\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") " pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:21.115772 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.115744    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:21.118093 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.118073    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls\") pod \"success-200-isvc-717c3-predictor-7db58fb9f4-htnjs\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") " pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:21.182025 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.181996    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:21.314106 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.314073    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"]
Jul 28 10:19:21.316971 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:19:21.316949    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96201bf8_4ea2_4c46_b607_c34fc7fc0fc6.slice/crio-6d65efe9113cc975bc2588aad369fd8c0c730b856990d6106f9307bef8b11549 WatchSource:0}: Error finding container 6d65efe9113cc975bc2588aad369fd8c0c730b856990d6106f9307bef8b11549: Status 404 returned error can't find the container with id 6d65efe9113cc975bc2588aad369fd8c0c730b856990d6106f9307bef8b11549
Jul 28 10:19:21.361935 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.361907    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" event={"ID":"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6","Type":"ContainerStarted","Data":"6d65efe9113cc975bc2588aad369fd8c0c730b856990d6106f9307bef8b11549"}
Jul 28 10:19:21.382119 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.382096    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:21.507799 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:21.507778    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"]
Jul 28 10:19:21.510249 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:19:21.510223    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb3a5c4_6a18_4047_8d18_dc66c7f47f7d.slice/crio-47c702c05cd4458d7a5cd037a05c3687ca2dafc38dfdb669c5365cf7aaaa8ced WatchSource:0}: Error finding container 47c702c05cd4458d7a5cd037a05c3687ca2dafc38dfdb669c5365cf7aaaa8ced: Status 404 returned error can't find the container with id 47c702c05cd4458d7a5cd037a05c3687ca2dafc38dfdb669c5365cf7aaaa8ced
Jul 28 10:19:22.380208 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:22.380075    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" event={"ID":"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d","Type":"ContainerStarted","Data":"47c702c05cd4458d7a5cd037a05c3687ca2dafc38dfdb669c5365cf7aaaa8ced"}
Jul 28 10:19:37.463168 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:37.463078    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" event={"ID":"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6","Type":"ContainerStarted","Data":"a018b9b9e704e14d111500131f07e43cfdfa331909b4a16ff1b02bf13a1776ae"}
Jul 28 10:19:37.465867 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:37.465796    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" event={"ID":"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d","Type":"ContainerStarted","Data":"0ac9884be112bb4bb39f057ee4bb55d0aa56788c77f15058643258f2817ee46d"}
Jul 28 10:19:39.475572 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.475507    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" event={"ID":"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d","Type":"ContainerStarted","Data":"031c6eb637c98b2d2fecf5395dc5c49630728dcbf196be60067b9d8dac2bdbf2"}
Jul 28 10:19:39.475572 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.475577    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:39.475994 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.475726    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:39.476938 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.476894    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 10:19:39.477290 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.477274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" event={"ID":"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6","Type":"ContainerStarted","Data":"3d2b3d544f30e5817ead06533b0b321db5972186d8d821cc4458fc97996b4238"}
Jul 28 10:19:39.477657 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.477642    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:39.477835 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.477816    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:39.478843 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.478792    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 10:19:39.507162 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:39.507117    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podStartSLOduration=1.6972360850000001 podStartE2EDuration="19.507104321s" totalImagesPullingTime="17.809868236s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:19:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:19:39.505665313 +0000 UTC m=+687.080858256" watchObservedRunningTime="2026-07-28 10:19:39.507104321 +0000 UTC m=+687.082297249"
Jul 28 10:19:40.483408 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:40.483373    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 10:19:40.483883 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:40.483447    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 10:19:41.487211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:41.487175    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 10:19:45.488266 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:45.488236    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:19:45.488848 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:45.488820    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 10:19:45.510201 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:45.510157    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podStartSLOduration=7.499886955 podStartE2EDuration="25.510140381s" totalImagesPullingTime="18.010253426s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:19:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:19:39.541227075 +0000 UTC m=+687.116420063" watchObservedRunningTime="2026-07-28 10:19:45.510140381 +0000 UTC m=+693.085333310"
Jul 28 10:19:46.491649 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:46.491622    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:19:46.492153 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:46.492128    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 10:19:55.489567 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:55.489531    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 10:19:56.492524 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:19:56.492489    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 10:20:05.489300 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:05.489262    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 10:20:06.492400 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:06.492358    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 10:20:15.489169 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:15.489129    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.20:8080: connect: connection refused"
Jul 28 10:20:16.492951 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:16.492913    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.21:8080: connect: connection refused"
Jul 28 10:20:25.489887 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:25.489855    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:20:26.495864 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:26.495835    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:20:50.383083 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.383048    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"]
Jul 28 10:20:50.383659 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.383365    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container" containerID="cri-o://0ac9884be112bb4bb39f057ee4bb55d0aa56788c77f15058643258f2817ee46d" gracePeriod=30
Jul 28 10:20:50.383659 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.383489    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kube-rbac-proxy" containerID="cri-o://031c6eb637c98b2d2fecf5395dc5c49630728dcbf196be60067b9d8dac2bdbf2" gracePeriod=30
Jul 28 10:20:50.440405 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.440365    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"]
Jul 28 10:20:50.440645 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.440618    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container" containerID="cri-o://a018b9b9e704e14d111500131f07e43cfdfa331909b4a16ff1b02bf13a1776ae" gracePeriod=30
Jul 28 10:20:50.440748 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.440687    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kube-rbac-proxy" containerID="cri-o://3d2b3d544f30e5817ead06533b0b321db5972186d8d821cc4458fc97996b4238" gracePeriod=30
Jul 28 10:20:50.445417 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.445388    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"]
Jul 28 10:20:50.448096 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.448079    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.450423 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.450404    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e92a5-predictor-serving-cert\""
Jul 28 10:20:50.450423 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.450416    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e92a5-kube-rbac-proxy-sar-config\""
Jul 28 10:20:50.458786 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.458764    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"]
Jul 28 10:20:50.480243 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.480217    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5lxj\" (UniqueName: \"kubernetes.io/projected/ed33011d-7843-43e9-9b70-8db094db7646-kube-api-access-z5lxj\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.480360 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.480296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.480431 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.480366    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed33011d-7843-43e9-9b70-8db094db7646-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.484350 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.484324    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.20:8643/healthz\": dial tcp 10.134.0.20:8643: connect: connection refused"
Jul 28 10:20:50.547037 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.547014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"]
Jul 28 10:20:50.549426 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.549404    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.551729 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.551693    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e92a5-predictor-serving-cert\""
Jul 28 10:20:50.551893 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.551752    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e92a5-kube-rbac-proxy-sar-config\""
Jul 28 10:20:50.558867 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.558841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"]
Jul 28 10:20:50.581239 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.581211    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5lxj\" (UniqueName: \"kubernetes.io/projected/ed33011d-7843-43e9-9b70-8db094db7646-kube-api-access-z5lxj\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.581362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.581245    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f5654d-0165-47e1-a1a1-ea27785211d5-proxy-tls\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.581362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.581276    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.581362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.581298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.581362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.581342    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szkds\" (UniqueName: \"kubernetes.io/projected/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-api-access-szkds\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.581362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.581361    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed33011d-7843-43e9-9b70-8db094db7646-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.581601 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:20:50.581368    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-serving-cert: secret "success-200-isvc-e92a5-predictor-serving-cert" not found
Jul 28 10:20:50.581601 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:20:50.581456    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls podName:ed33011d-7843-43e9-9b70-8db094db7646 nodeName:}" failed. No retries permitted until 2026-07-28 10:20:51.08142989 +0000 UTC m=+758.656622813 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls") pod "success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" (UID: "ed33011d-7843-43e9-9b70-8db094db7646") : secret "success-200-isvc-e92a5-predictor-serving-cert" not found
Jul 28 10:20:50.582044 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.582024    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed33011d-7843-43e9-9b70-8db094db7646-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.593930 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.593909    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5lxj\" (UniqueName: \"kubernetes.io/projected/ed33011d-7843-43e9-9b70-8db094db7646-kube-api-access-z5lxj\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:50.682277 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.682197    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.682277 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.682258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szkds\" (UniqueName: \"kubernetes.io/projected/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-api-access-szkds\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.682497 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.682311    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f5654d-0165-47e1-a1a1-ea27785211d5-proxy-tls\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.682869 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.682848    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.684752 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.684704    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f5654d-0165-47e1-a1a1-ea27785211d5-proxy-tls\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.690961 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.690937    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szkds\" (UniqueName: \"kubernetes.io/projected/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-api-access-szkds\") pod \"error-404-isvc-e92a5-predictor-599bf66b77-hrnbd\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") " pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.741601 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.741569    2560 generic.go:356] "Generic (PLEG): container finished" podID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerID="031c6eb637c98b2d2fecf5395dc5c49630728dcbf196be60067b9d8dac2bdbf2" exitCode=2
Jul 28 10:20:50.741780 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.741649    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" event={"ID":"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d","Type":"ContainerDied","Data":"031c6eb637c98b2d2fecf5395dc5c49630728dcbf196be60067b9d8dac2bdbf2"}
Jul 28 10:20:50.743698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.743670    2560 generic.go:356] "Generic (PLEG): container finished" podID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerID="3d2b3d544f30e5817ead06533b0b321db5972186d8d821cc4458fc97996b4238" exitCode=2
Jul 28 10:20:50.743833 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.743706    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" event={"ID":"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6","Type":"ContainerDied","Data":"3d2b3d544f30e5817ead06533b0b321db5972186d8d821cc4458fc97996b4238"}
Jul 28 10:20:50.860942 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.860911    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:50.989778 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:50.989757    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"]
Jul 28 10:20:50.992061 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:20:50.992035    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98f5654d_0165_47e1_a1a1_ea27785211d5.slice/crio-d6a79de52b2fa412cfd19ef7667ec1efab1360a4ae4bea11a8405e7790410cdd WatchSource:0}: Error finding container d6a79de52b2fa412cfd19ef7667ec1efab1360a4ae4bea11a8405e7790410cdd: Status 404 returned error can't find the container with id d6a79de52b2fa412cfd19ef7667ec1efab1360a4ae4bea11a8405e7790410cdd
Jul 28 10:20:51.085924 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.085899    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:51.088207 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.088188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls\") pod \"success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") " pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:51.364333 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.364254    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:51.487976 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.487937    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.21:8643/healthz\": dial tcp 10.134.0.21:8643: connect: connection refused"
Jul 28 10:20:51.493979 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.493852    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"]
Jul 28 10:20:51.496520 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:20:51.496495    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded33011d_7843_43e9_9b70_8db094db7646.slice/crio-49b3f684625ec1b768b07d52fc21673a040b6d81e4010bff5de2bbc8889ccab1 WatchSource:0}: Error finding container 49b3f684625ec1b768b07d52fc21673a040b6d81e4010bff5de2bbc8889ccab1: Status 404 returned error can't find the container with id 49b3f684625ec1b768b07d52fc21673a040b6d81e4010bff5de2bbc8889ccab1
Jul 28 10:20:51.751301 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.751264    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" event={"ID":"ed33011d-7843-43e9-9b70-8db094db7646","Type":"ContainerStarted","Data":"0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b"}
Jul 28 10:20:51.751511 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.751345    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" event={"ID":"ed33011d-7843-43e9-9b70-8db094db7646","Type":"ContainerStarted","Data":"5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63"}
Jul 28 10:20:51.751511 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.751384    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:51.751511 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.751409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" event={"ID":"ed33011d-7843-43e9-9b70-8db094db7646","Type":"ContainerStarted","Data":"49b3f684625ec1b768b07d52fc21673a040b6d81e4010bff5de2bbc8889ccab1"}
Jul 28 10:20:51.753223 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.753196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" event={"ID":"98f5654d-0165-47e1-a1a1-ea27785211d5","Type":"ContainerStarted","Data":"e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9"}
Jul 28 10:20:51.753362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.753258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" event={"ID":"98f5654d-0165-47e1-a1a1-ea27785211d5","Type":"ContainerStarted","Data":"021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad"}
Jul 28 10:20:51.753362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.753291    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" event={"ID":"98f5654d-0165-47e1-a1a1-ea27785211d5","Type":"ContainerStarted","Data":"d6a79de52b2fa412cfd19ef7667ec1efab1360a4ae4bea11a8405e7790410cdd"}
Jul 28 10:20:51.753362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.753358    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:51.753497 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.753389    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:51.754753 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.754669    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 10:20:51.768769 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.768703    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podStartSLOduration=1.76868945 podStartE2EDuration="1.76868945s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:20:51.767357296 +0000 UTC m=+759.342550226" watchObservedRunningTime="2026-07-28 10:20:51.76868945 +0000 UTC m=+759.343882381"
Jul 28 10:20:51.785397 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:51.785348    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podStartSLOduration=1.785334786 podStartE2EDuration="1.785334786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:20:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:20:51.783556688 +0000 UTC m=+759.358749616" watchObservedRunningTime="2026-07-28 10:20:51.785334786 +0000 UTC m=+759.360527715"
Jul 28 10:20:52.757730 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:52.757679    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 10:20:52.758177 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:52.757833    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:52.758880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:52.758856    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 10:20:53.764038 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.764007    2560 generic.go:356] "Generic (PLEG): container finished" podID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerID="a018b9b9e704e14d111500131f07e43cfdfa331909b4a16ff1b02bf13a1776ae" exitCode=0
Jul 28 10:20:53.764432 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.764077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" event={"ID":"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6","Type":"ContainerDied","Data":"a018b9b9e704e14d111500131f07e43cfdfa331909b4a16ff1b02bf13a1776ae"}
Jul 28 10:20:53.766043 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.766002    2560 generic.go:356] "Generic (PLEG): container finished" podID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerID="0ac9884be112bb4bb39f057ee4bb55d0aa56788c77f15058643258f2817ee46d" exitCode=0
Jul 28 10:20:53.766146 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.766062    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" event={"ID":"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d","Type":"ContainerDied","Data":"0ac9884be112bb4bb39f057ee4bb55d0aa56788c77f15058643258f2817ee46d"}
Jul 28 10:20:53.766405 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.766369    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 10:20:53.831483 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.831461    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:20:53.881630 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.881611    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:20:53.910212 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.910142    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-rbac-proxy-sar-config\") pod \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") "
Jul 28 10:20:53.910345 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.910214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls\") pod \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") "
Jul 28 10:20:53.910345 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.910267    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvn7s\" (UniqueName: \"kubernetes.io/projected/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-api-access-wvn7s\") pod \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\" (UID: \"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d\") "
Jul 28 10:20:53.910532 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.910508    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" (UID: "ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:20:53.912380 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.912359    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" (UID: "ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:20:53.912458 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:53.912405    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-api-access-wvn7s" (OuterVolumeSpecName: "kube-api-access-wvn7s") pod "ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" (UID: "ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d"). InnerVolumeSpecName "kube-api-access-wvn7s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:20:54.011598 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.011554    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-proxy-tls\") pod \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") "
Jul 28 10:20:54.011598 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.011616    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rqvs4\" (UniqueName: \"kubernetes.io/projected/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-api-access-rqvs4\") pod \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") "
Jul 28 10:20:54.011866 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.011798    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-rbac-proxy-sar-config\") pod \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\" (UID: \"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6\") "
Jul 28 10:20:54.012037 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.012015    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvn7s\" (UniqueName: \"kubernetes.io/projected/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-api-access-wvn7s\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:20:54.012037 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.012036    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:20:54.012164 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.012048    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:20:54.012164 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.012097    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" (UID: "96201bf8-4ea2-4c46-b607-c34fc7fc0fc6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:20:54.013643 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.013623    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-api-access-rqvs4" (OuterVolumeSpecName: "kube-api-access-rqvs4") pod "96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" (UID: "96201bf8-4ea2-4c46-b607-c34fc7fc0fc6"). InnerVolumeSpecName "kube-api-access-rqvs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:20:54.013643 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.013637    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" (UID: "96201bf8-4ea2-4c46-b607-c34fc7fc0fc6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:20:54.113356 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.113323    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqvs4\" (UniqueName: \"kubernetes.io/projected/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-api-access-rqvs4\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:20:54.113356 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.113351    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:20:54.113356 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.113361    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:20:54.771490 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.771466    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"
Jul 28 10:20:54.771970 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.771501    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw" event={"ID":"96201bf8-4ea2-4c46-b607-c34fc7fc0fc6","Type":"ContainerDied","Data":"6d65efe9113cc975bc2588aad369fd8c0c730b856990d6106f9307bef8b11549"}
Jul 28 10:20:54.771970 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.771570    2560 scope.go:117] "RemoveContainer" containerID="3d2b3d544f30e5817ead06533b0b321db5972186d8d821cc4458fc97996b4238"
Jul 28 10:20:54.773207 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.773182    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs" event={"ID":"ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d","Type":"ContainerDied","Data":"47c702c05cd4458d7a5cd037a05c3687ca2dafc38dfdb669c5365cf7aaaa8ced"}
Jul 28 10:20:54.773323 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.773193    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"
Jul 28 10:20:54.780356 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.780320    2560 scope.go:117] "RemoveContainer" containerID="a018b9b9e704e14d111500131f07e43cfdfa331909b4a16ff1b02bf13a1776ae"
Jul 28 10:20:54.788019 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.788001    2560 scope.go:117] "RemoveContainer" containerID="031c6eb637c98b2d2fecf5395dc5c49630728dcbf196be60067b9d8dac2bdbf2"
Jul 28 10:20:54.795302 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.795280    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"]
Jul 28 10:20:54.795670 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.795658    2560 scope.go:117] "RemoveContainer" containerID="0ac9884be112bb4bb39f057ee4bb55d0aa56788c77f15058643258f2817ee46d"
Jul 28 10:20:54.798919 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.798898    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-717c3-predictor-67f5fb9455-8qnmw"]
Jul 28 10:20:54.811211 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.811190    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"]
Jul 28 10:20:54.815796 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.815777    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-717c3-predictor-7db58fb9f4-htnjs"]
Jul 28 10:20:54.979566 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.979474    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" path="/var/lib/kubelet/pods/96201bf8-4ea2-4c46-b607-c34fc7fc0fc6/volumes"
Jul 28 10:20:54.980001 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:54.979981    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" path="/var/lib/kubelet/pods/ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d/volumes"
Jul 28 10:20:57.762345 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:57.762318    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:20:57.762743 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:57.762700    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 10:20:58.771116 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:58.771091    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:20:58.771602 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:20:58.771576    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 10:21:07.763567 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:07.763530    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 10:21:08.771608 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:08.771567    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 10:21:17.762951 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:17.762914    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 10:21:18.771904 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:18.771860    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 10:21:27.763038 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:27.762998    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 10:21:28.771627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:28.771586    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.22:8080: connect: connection refused"
Jul 28 10:21:37.763478 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:37.763407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:21:38.771905 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:38.771873    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:21:40.458777 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.458744    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"]
Jul 28 10:21:40.459241 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459195    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kube-rbac-proxy"
Jul 28 10:21:40.459241 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459213    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kube-rbac-proxy"
Jul 28 10:21:40.459375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459244    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kube-rbac-proxy"
Jul 28 10:21:40.459375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459253    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kube-rbac-proxy"
Jul 28 10:21:40.459375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459272    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container"
Jul 28 10:21:40.459375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459281    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container"
Jul 28 10:21:40.459375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459292    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container"
Jul 28 10:21:40.459375 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459300    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container"
Jul 28 10:21:40.459646 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459382    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kserve-container"
Jul 28 10:21:40.459646 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459397    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kube-rbac-proxy"
Jul 28 10:21:40.459646 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459408    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="96201bf8-4ea2-4c46-b607-c34fc7fc0fc6" containerName="kube-rbac-proxy"
Jul 28 10:21:40.459646 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.459420    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ebb3a5c4-6a18-4047-8d18-dc66c7f47f7d" containerName="kserve-container"
Jul 28 10:21:40.461518 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.461497    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:40.463593 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.463568    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0b9d4-predictor-serving-cert\""
Jul 28 10:21:40.463677 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.463599    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-0b9d4-kube-rbac-proxy-sar-config\""
Jul 28 10:21:40.470871 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.470848    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"]
Jul 28 10:21:40.557425 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.557391    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"]
Jul 28 10:21:40.559707 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.559689    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.561755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.561732    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0b9d4-predictor-serving-cert\""
Jul 28 10:21:40.561882 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.561738    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-0b9d4-kube-rbac-proxy-sar-config\""
Jul 28 10:21:40.565478 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.565451    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbwz9\" (UniqueName: \"kubernetes.io/projected/21380bc3-7d5d-4d24-987d-6013ac575626-kube-api-access-tbwz9\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:40.565569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.565546    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:40.565612 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.565593    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.565659 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.565617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21380bc3-7d5d-4d24-987d-6013ac575626-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:40.565659 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.565650    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfxfq\" (UniqueName: \"kubernetes.io/projected/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-api-access-pfxfq\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.565770 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.565710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.568848 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.568828    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"]
Jul 28 10:21:40.667016 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.666986    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfxfq\" (UniqueName: \"kubernetes.io/projected/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-api-access-pfxfq\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.667192 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.667023    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.667192 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.667051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbwz9\" (UniqueName: \"kubernetes.io/projected/21380bc3-7d5d-4d24-987d-6013ac575626-kube-api-access-tbwz9\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:40.667192 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.667095    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:40.667192 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.667138    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.667192 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.667167    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21380bc3-7d5d-4d24-987d-6013ac575626-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:40.667433 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:21:40.667237    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-serving-cert: secret "success-200-isvc-0b9d4-predictor-serving-cert" not found
Jul 28 10:21:40.667433 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:21:40.667271    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-serving-cert: secret "error-404-isvc-0b9d4-predictor-serving-cert" not found
Jul 28 10:21:40.667433 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:21:40.667298    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls podName:21380bc3-7d5d-4d24-987d-6013ac575626 nodeName:}" failed. No retries permitted until 2026-07-28 10:21:41.167278761 +0000 UTC m=+808.742471681 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls") pod "success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" (UID: "21380bc3-7d5d-4d24-987d-6013ac575626") : secret "success-200-isvc-0b9d4-predictor-serving-cert" not found
Jul 28 10:21:40.667433 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:21:40.667349    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls podName:5493527c-3b35-4768-aa56-fbbcc9688dc3 nodeName:}" failed. No retries permitted until 2026-07-28 10:21:41.16732477 +0000 UTC m=+808.742517705 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls") pod "error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" (UID: "5493527c-3b35-4768-aa56-fbbcc9688dc3") : secret "error-404-isvc-0b9d4-predictor-serving-cert" not found
Jul 28 10:21:40.667908 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.667888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.667954 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.667888    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21380bc3-7d5d-4d24-987d-6013ac575626-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:40.679525 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.679501    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfxfq\" (UniqueName: \"kubernetes.io/projected/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-api-access-pfxfq\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:40.679633 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:40.679504    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbwz9\" (UniqueName: \"kubernetes.io/projected/21380bc3-7d5d-4d24-987d-6013ac575626-kube-api-access-tbwz9\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:41.171475 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.171443    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:41.171647 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.171509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:41.171647 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:21:41.171589    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-serving-cert: secret "error-404-isvc-0b9d4-predictor-serving-cert" not found
Jul 28 10:21:41.171759 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:21:41.171652    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls podName:5493527c-3b35-4768-aa56-fbbcc9688dc3 nodeName:}" failed. No retries permitted until 2026-07-28 10:21:42.171637777 +0000 UTC m=+809.746830684 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls") pod "error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" (UID: "5493527c-3b35-4768-aa56-fbbcc9688dc3") : secret "error-404-isvc-0b9d4-predictor-serving-cert" not found
Jul 28 10:21:41.173725 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.173701    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls\") pod \"success-200-isvc-0b9d4-predictor-54b67cb57d-48n99\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") " pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:41.373236 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.373203    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:41.498450 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.498427    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"]
Jul 28 10:21:41.500684 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:21:41.500660    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21380bc3_7d5d_4d24_987d_6013ac575626.slice/crio-bc1ec366be47389edab51639759b24326d859e477c4a53917f08e12ba10dd369 WatchSource:0}: Error finding container bc1ec366be47389edab51639759b24326d859e477c4a53917f08e12ba10dd369: Status 404 returned error can't find the container with id bc1ec366be47389edab51639759b24326d859e477c4a53917f08e12ba10dd369
Jul 28 10:21:41.964308 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.964276    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" event={"ID":"21380bc3-7d5d-4d24-987d-6013ac575626","Type":"ContainerStarted","Data":"9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac"}
Jul 28 10:21:41.964493 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.964359    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:41.964493 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.964391    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:41.964493 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.964415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" event={"ID":"21380bc3-7d5d-4d24-987d-6013ac575626","Type":"ContainerStarted","Data":"806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508"}
Jul 28 10:21:41.964493 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.964443    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" event={"ID":"21380bc3-7d5d-4d24-987d-6013ac575626","Type":"ContainerStarted","Data":"bc1ec366be47389edab51639759b24326d859e477c4a53917f08e12ba10dd369"}
Jul 28 10:21:41.965891 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.965865    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:21:41.981565 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:41.981519    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podStartSLOduration=1.981505632 podStartE2EDuration="1.981505632s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:21:41.979988644 +0000 UTC m=+809.555181584" watchObservedRunningTime="2026-07-28 10:21:41.981505632 +0000 UTC m=+809.556698562"
Jul 28 10:21:42.177690 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.177652    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:42.180021 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.179992    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls\") pod \"error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") " pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:42.371088 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.371009    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:42.498067 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.498041    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"]
Jul 28 10:21:42.500601 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:21:42.500572    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5493527c_3b35_4768_aa56_fbbcc9688dc3.slice/crio-f69cac0a663ea4a54a1e5eb540bbdc2ae2951d53f5fbcd4069a321db6494c72a WatchSource:0}: Error finding container f69cac0a663ea4a54a1e5eb540bbdc2ae2951d53f5fbcd4069a321db6494c72a: Status 404 returned error can't find the container with id f69cac0a663ea4a54a1e5eb540bbdc2ae2951d53f5fbcd4069a321db6494c72a
Jul 28 10:21:42.970229 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.970188    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" event={"ID":"5493527c-3b35-4768-aa56-fbbcc9688dc3","Type":"ContainerStarted","Data":"e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76"}
Jul 28 10:21:42.970373 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.970260    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" event={"ID":"5493527c-3b35-4768-aa56-fbbcc9688dc3","Type":"ContainerStarted","Data":"be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4"}
Jul 28 10:21:42.970373 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.970332    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" event={"ID":"5493527c-3b35-4768-aa56-fbbcc9688dc3","Type":"ContainerStarted","Data":"f69cac0a663ea4a54a1e5eb540bbdc2ae2951d53f5fbcd4069a321db6494c72a"}
Jul 28 10:21:42.970610 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.970579    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:21:42.988067 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:42.988025    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podStartSLOduration=2.988010905 podStartE2EDuration="2.988010905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:21:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:21:42.986475668 +0000 UTC m=+810.561668634" watchObservedRunningTime="2026-07-28 10:21:42.988010905 +0000 UTC m=+810.563203909"
Jul 28 10:21:43.974401 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:43.974372    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:43.974833 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:43.974432    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:43.975514 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:43.975486    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 10:21:44.978583 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:44.978548    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 10:21:47.975749 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:47.975390    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:21:47.980236 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:47.980202    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:21:49.982819 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:49.982789    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:21:49.983247 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:49.983221    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 10:21:57.976736 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:57.976673    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:21:59.983882 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:21:59.983842    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 10:22:07.976735 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:22:07.976673    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:22:09.983860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:22:09.983824    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 10:22:17.976524 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:22:17.976488    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:22:19.983759 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:22:19.983700    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.25:8080: connect: connection refused"
Jul 28 10:22:27.977341 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:22:27.977312    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:22:29.983875 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:22:29.983848    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:23:12.867059 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:23:12.867030    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:23:12.868421 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:23:12.868390    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:28:12.891388 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:28:12.891351    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:28:12.891857 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:28:12.891351    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:30:05.276992 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.276951    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"]
Jul 28 10:30:05.277572 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.277251    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container" containerID="cri-o://5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63" gracePeriod=30
Jul 28 10:30:05.277572 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.277346    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kube-rbac-proxy" containerID="cri-o://0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b" gracePeriod=30
Jul 28 10:30:05.398141 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.398110    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"]
Jul 28 10:30:05.401672 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.401653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.403898 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.403869    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-af89d-predictor-serving-cert\""
Jul 28 10:30:05.404028 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.403903    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-af89d-kube-rbac-proxy-sar-config\""
Jul 28 10:30:05.412082 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.412056    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"]
Jul 28 10:30:05.459700 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.459664    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"]
Jul 28 10:30:05.463116 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.463092    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.465148 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.465125    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-af89d-predictor-serving-cert\""
Jul 28 10:30:05.465379 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.465256    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-af89d-kube-rbac-proxy-sar-config\""
Jul 28 10:30:05.473098 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.473073    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"]
Jul 28 10:30:05.507608 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.507577    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"]
Jul 28 10:30:05.513326 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.508219    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container" containerID="cri-o://021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad" gracePeriod=30
Jul 28 10:30:05.513326 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.508504    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kube-rbac-proxy" containerID="cri-o://e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9" gracePeriod=30
Jul 28 10:30:05.541342 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.541263    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7l9l\" (UniqueName: \"kubernetes.io/projected/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-api-access-f7l9l\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.541342 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.541300    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.541342 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.541319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t5qf\" (UniqueName: \"kubernetes.io/projected/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-api-access-6t5qf\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.541582 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.541360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ba1909-9221-445a-8ed6-4b277f54fcb4-proxy-tls\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.541582 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.541396    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.541582 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.541430    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8416209-8c09-4f1c-a627-65de821b7f0c-proxy-tls\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.642366 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.642325    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.642509 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.642370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8416209-8c09-4f1c-a627-65de821b7f0c-proxy-tls\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.642509 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.642410    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7l9l\" (UniqueName: \"kubernetes.io/projected/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-api-access-f7l9l\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.642509 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.642428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.642509 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.642444    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t5qf\" (UniqueName: \"kubernetes.io/projected/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-api-access-6t5qf\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.642509 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.642471    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ba1909-9221-445a-8ed6-4b277f54fcb4-proxy-tls\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.643281 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.643156    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.643442 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.643319    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.644970 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.644944    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8416209-8c09-4f1c-a627-65de821b7f0c-proxy-tls\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.645070 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.645048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ba1909-9221-445a-8ed6-4b277f54fcb4-proxy-tls\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.651439 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.651413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7l9l\" (UniqueName: \"kubernetes.io/projected/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-api-access-f7l9l\") pod \"error-404-isvc-af89d-predictor-6bff89d85-j66sp\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") " pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.651554 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.651527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t5qf\" (UniqueName: \"kubernetes.io/projected/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-api-access-6t5qf\") pod \"success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") " pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.715386 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.715338    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:05.773616 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.773587    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:05.853560 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.853527    2560 generic.go:356] "Generic (PLEG): container finished" podID="ed33011d-7843-43e9-9b70-8db094db7646" containerID="0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b" exitCode=2
Jul 28 10:30:05.853753 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.853606    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" event={"ID":"ed33011d-7843-43e9-9b70-8db094db7646","Type":"ContainerDied","Data":"0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b"}
Jul 28 10:30:05.856120 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.855787    2560 generic.go:356] "Generic (PLEG): container finished" podID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerID="e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9" exitCode=2
Jul 28 10:30:05.856120 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.855836    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" event={"ID":"98f5654d-0165-47e1-a1a1-ea27785211d5","Type":"ContainerDied","Data":"e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9"}
Jul 28 10:30:05.858017 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.857669    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"]
Jul 28 10:30:05.860069 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:30:05.860040    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode3ba1909_9221_445a_8ed6_4b277f54fcb4.slice/crio-a9146dad3855cc106bad6a95119801e0d3d117037d186b526e33841db5b1464a WatchSource:0}: Error finding container a9146dad3855cc106bad6a95119801e0d3d117037d186b526e33841db5b1464a: Status 404 returned error can't find the container with id a9146dad3855cc106bad6a95119801e0d3d117037d186b526e33841db5b1464a
Jul 28 10:30:05.908696 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:05.908673    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"]
Jul 28 10:30:05.911202 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:30:05.911177    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8416209_8c09_4f1c_a627_65de821b7f0c.slice/crio-097b45e649ed0cb14584ba5f662f3e36bcb70e91b083c6eb057e3636436c7af0 WatchSource:0}: Error finding container 097b45e649ed0cb14584ba5f662f3e36bcb70e91b083c6eb057e3636436c7af0: Status 404 returned error can't find the container with id 097b45e649ed0cb14584ba5f662f3e36bcb70e91b083c6eb057e3636436c7af0
Jul 28 10:30:06.861784 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.861710    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" event={"ID":"e3ba1909-9221-445a-8ed6-4b277f54fcb4","Type":"ContainerStarted","Data":"e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6"}
Jul 28 10:30:06.862245 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.861799    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" event={"ID":"e3ba1909-9221-445a-8ed6-4b277f54fcb4","Type":"ContainerStarted","Data":"0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f"}
Jul 28 10:30:06.862245 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.861839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:06.862245 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.861867    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" event={"ID":"e3ba1909-9221-445a-8ed6-4b277f54fcb4","Type":"ContainerStarted","Data":"a9146dad3855cc106bad6a95119801e0d3d117037d186b526e33841db5b1464a"}
Jul 28 10:30:06.862245 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.861900    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:06.863546 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.863516    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:30:06.863792 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.863767    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" event={"ID":"a8416209-8c09-4f1c-a627-65de821b7f0c","Type":"ContainerStarted","Data":"49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c"}
Jul 28 10:30:06.863855 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.863819    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" event={"ID":"a8416209-8c09-4f1c-a627-65de821b7f0c","Type":"ContainerStarted","Data":"3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557"}
Jul 28 10:30:06.863855 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.863841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" event={"ID":"a8416209-8c09-4f1c-a627-65de821b7f0c","Type":"ContainerStarted","Data":"097b45e649ed0cb14584ba5f662f3e36bcb70e91b083c6eb057e3636436c7af0"}
Jul 28 10:30:06.863934 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.863863    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:06.863934 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.863914    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:06.864951 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.864929    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:30:06.882240 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.882197    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podStartSLOduration=1.882184842 podStartE2EDuration="1.882184842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:30:06.880960571 +0000 UTC m=+1314.456153499" watchObservedRunningTime="2026-07-28 10:30:06.882184842 +0000 UTC m=+1314.457377772"
Jul 28 10:30:06.902302 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:06.902263    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podStartSLOduration=1.902250477 podStartE2EDuration="1.902250477s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:30:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:30:06.900345915 +0000 UTC m=+1314.475538842" watchObservedRunningTime="2026-07-28 10:30:06.902250477 +0000 UTC m=+1314.477443406"
Jul 28 10:30:07.757876 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:07.757841    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.23:8643/healthz\": dial tcp 10.134.0.23:8643: connect: connection refused"
Jul 28 10:30:07.763250 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:07.763225    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.23:8080: connect: connection refused"
Jul 28 10:30:07.868083 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:07.868046    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:30:07.868494 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:07.868158    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:30:08.527178 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.527157    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:30:08.670010 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.669983    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls\") pod \"ed33011d-7843-43e9-9b70-8db094db7646\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") "
Jul 28 10:30:08.670105 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.670019    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5lxj\" (UniqueName: \"kubernetes.io/projected/ed33011d-7843-43e9-9b70-8db094db7646-kube-api-access-z5lxj\") pod \"ed33011d-7843-43e9-9b70-8db094db7646\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") "
Jul 28 10:30:08.670105 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.670075    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed33011d-7843-43e9-9b70-8db094db7646-kube-rbac-proxy-sar-config\") pod \"ed33011d-7843-43e9-9b70-8db094db7646\" (UID: \"ed33011d-7843-43e9-9b70-8db094db7646\") "
Jul 28 10:30:08.670488 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.670450    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed33011d-7843-43e9-9b70-8db094db7646-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ed33011d-7843-43e9-9b70-8db094db7646" (UID: "ed33011d-7843-43e9-9b70-8db094db7646"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:30:08.671952 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.671928    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ed33011d-7843-43e9-9b70-8db094db7646" (UID: "ed33011d-7843-43e9-9b70-8db094db7646"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:30:08.672108 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.672086    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed33011d-7843-43e9-9b70-8db094db7646-kube-api-access-z5lxj" (OuterVolumeSpecName: "kube-api-access-z5lxj") pod "ed33011d-7843-43e9-9b70-8db094db7646" (UID: "ed33011d-7843-43e9-9b70-8db094db7646"). InnerVolumeSpecName "kube-api-access-z5lxj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:30:08.770790 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.770760    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ed33011d-7843-43e9-9b70-8db094db7646-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:08.770790 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.770788    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ed33011d-7843-43e9-9b70-8db094db7646-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:08.770790 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.770799    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5lxj\" (UniqueName: \"kubernetes.io/projected/ed33011d-7843-43e9-9b70-8db094db7646-kube-api-access-z5lxj\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:08.850975 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.850954    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:30:08.873762 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.873708    2560 generic.go:356] "Generic (PLEG): container finished" podID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerID="021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad" exitCode=0
Jul 28 10:30:08.874134 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.873751    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" event={"ID":"98f5654d-0165-47e1-a1a1-ea27785211d5","Type":"ContainerDied","Data":"021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad"}
Jul 28 10:30:08.874134 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.873771    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"
Jul 28 10:30:08.874134 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.873869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd" event={"ID":"98f5654d-0165-47e1-a1a1-ea27785211d5","Type":"ContainerDied","Data":"d6a79de52b2fa412cfd19ef7667ec1efab1360a4ae4bea11a8405e7790410cdd"}
Jul 28 10:30:08.874134 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.873914    2560 scope.go:117] "RemoveContainer" containerID="e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9"
Jul 28 10:30:08.875728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.875688    2560 generic.go:356] "Generic (PLEG): container finished" podID="ed33011d-7843-43e9-9b70-8db094db7646" containerID="5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63" exitCode=0
Jul 28 10:30:08.875835 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.875764    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" event={"ID":"ed33011d-7843-43e9-9b70-8db094db7646","Type":"ContainerDied","Data":"5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63"}
Jul 28 10:30:08.875835 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.875773    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"
Jul 28 10:30:08.875835 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.875817    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn" event={"ID":"ed33011d-7843-43e9-9b70-8db094db7646","Type":"ContainerDied","Data":"49b3f684625ec1b768b07d52fc21673a040b6d81e4010bff5de2bbc8889ccab1"}
Jul 28 10:30:08.882747 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.882710    2560 scope.go:117] "RemoveContainer" containerID="021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad"
Jul 28 10:30:08.891500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.891486    2560 scope.go:117] "RemoveContainer" containerID="e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9"
Jul 28 10:30:08.891778 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:08.891752    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9\": container with ID starting with e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9 not found: ID does not exist" containerID="e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9"
Jul 28 10:30:08.891845 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.891789    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9"} err="failed to get container status \"e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9\": rpc error: code = NotFound desc = could not find container \"e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9\": container with ID starting with e17eb5f2fd60d374e9a230d2055ebd1cff4804627d5d133e2ae774e2fa906bf9 not found: ID does not exist"
Jul 28 10:30:08.891845 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.891813    2560 scope.go:117] "RemoveContainer" containerID="021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad"
Jul 28 10:30:08.892075 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:08.892059    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad\": container with ID starting with 021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad not found: ID does not exist" containerID="021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad"
Jul 28 10:30:08.892136 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.892079    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad"} err="failed to get container status \"021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad\": rpc error: code = NotFound desc = could not find container \"021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad\": container with ID starting with 021a0c7415fb332f4cd5045b5ccb29691a648fcffb498c5a7fc746ad830688ad not found: ID does not exist"
Jul 28 10:30:08.892136 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.892093    2560 scope.go:117] "RemoveContainer" containerID="0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b"
Jul 28 10:30:08.898828 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.898811    2560 scope.go:117] "RemoveContainer" containerID="5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63"
Jul 28 10:30:08.900996 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.900981    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"]
Jul 28 10:30:08.905833 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.905814    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e92a5-predictor-5cdccb45b8-bnrjn"]
Jul 28 10:30:08.906241 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.906224    2560 scope.go:117] "RemoveContainer" containerID="0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b"
Jul 28 10:30:08.906483 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:08.906465    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b\": container with ID starting with 0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b not found: ID does not exist" containerID="0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b"
Jul 28 10:30:08.906561 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.906495    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b"} err="failed to get container status \"0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b\": rpc error: code = NotFound desc = could not find container \"0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b\": container with ID starting with 0f59526993b029a8e8edafed6b1a881c1f7d10eb636a7b37cce1b7efe69d931b not found: ID does not exist"
Jul 28 10:30:08.906561 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.906517    2560 scope.go:117] "RemoveContainer" containerID="5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63"
Jul 28 10:30:08.906782 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:08.906765    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63\": container with ID starting with 5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63 not found: ID does not exist" containerID="5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63"
Jul 28 10:30:08.906820 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.906789    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63"} err="failed to get container status \"5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63\": rpc error: code = NotFound desc = could not find container \"5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63\": container with ID starting with 5fb7aa00ddce93841c988d229827991e2761022ba48dff1c7b1966a7d0bc9c63 not found: ID does not exist"
Jul 28 10:30:08.973223 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.973204    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szkds\" (UniqueName: \"kubernetes.io/projected/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-api-access-szkds\") pod \"98f5654d-0165-47e1-a1a1-ea27785211d5\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") "
Jul 28 10:30:08.973323 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.973237    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-rbac-proxy-sar-config\") pod \"98f5654d-0165-47e1-a1a1-ea27785211d5\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") "
Jul 28 10:30:08.973323 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.973281    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f5654d-0165-47e1-a1a1-ea27785211d5-proxy-tls\") pod \"98f5654d-0165-47e1-a1a1-ea27785211d5\" (UID: \"98f5654d-0165-47e1-a1a1-ea27785211d5\") "
Jul 28 10:30:08.973615 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.973594    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "98f5654d-0165-47e1-a1a1-ea27785211d5" (UID: "98f5654d-0165-47e1-a1a1-ea27785211d5"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:30:08.975190 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.975168    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-api-access-szkds" (OuterVolumeSpecName: "kube-api-access-szkds") pod "98f5654d-0165-47e1-a1a1-ea27785211d5" (UID: "98f5654d-0165-47e1-a1a1-ea27785211d5"). InnerVolumeSpecName "kube-api-access-szkds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:30:08.975257 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.975195    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98f5654d-0165-47e1-a1a1-ea27785211d5-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "98f5654d-0165-47e1-a1a1-ea27785211d5" (UID: "98f5654d-0165-47e1-a1a1-ea27785211d5"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:30:08.978413 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:08.978394    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed33011d-7843-43e9-9b70-8db094db7646" path="/var/lib/kubelet/pods/ed33011d-7843-43e9-9b70-8db094db7646/volumes"
Jul 28 10:30:09.074075 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:09.074049    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/98f5654d-0165-47e1-a1a1-ea27785211d5-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.074075 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:09.074070    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szkds\" (UniqueName: \"kubernetes.io/projected/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-api-access-szkds\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.074195 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:09.074079    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/98f5654d-0165-47e1-a1a1-ea27785211d5-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:09.191417 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:09.191353    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"]
Jul 28 10:30:09.195749 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:09.195728    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e92a5-predictor-599bf66b77-hrnbd"]
Jul 28 10:30:10.977627 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:10.977595    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" path="/var/lib/kubelet/pods/98f5654d-0165-47e1-a1a1-ea27785211d5/volumes"
Jul 28 10:30:12.874765 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:12.874559    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:12.875319 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:12.875288    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:30:12.875593 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:12.875575    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:12.876009 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:12.875989    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:30:22.875161 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:22.875125    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:30:22.876657 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:22.876632    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:30:32.875976 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:32.875934    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:30:32.876488 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:32.875934    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:30:42.875848 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:42.875764    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.27:8080: connect: connection refused"
Jul 28 10:30:42.876468 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:42.876026    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.26:8080: connect: connection refused"
Jul 28 10:30:52.875868 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:52.875832    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:30:52.876585 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:52.876567    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:30:55.110573 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.110533    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"]
Jul 28 10:30:55.111065 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.110822    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container" containerID="cri-o://806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508" gracePeriod=30
Jul 28 10:30:55.111065 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.110877    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kube-rbac-proxy" containerID="cri-o://9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac" gracePeriod=30
Jul 28 10:30:55.171625 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.171586    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"]
Jul 28 10:30:55.171878 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.171849    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container" containerID="cri-o://be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4" gracePeriod=30
Jul 28 10:30:55.171878 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.171864    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kube-rbac-proxy" containerID="cri-o://e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76" gracePeriod=30
Jul 28 10:30:55.185683 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.185658    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"]
Jul 28 10:30:55.186017 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186002    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kube-rbac-proxy"
Jul 28 10:30:55.186090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186020    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kube-rbac-proxy"
Jul 28 10:30:55.186090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186038    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container"
Jul 28 10:30:55.186090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186043    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container"
Jul 28 10:30:55.186090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186051    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kube-rbac-proxy"
Jul 28 10:30:55.186090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186056    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kube-rbac-proxy"
Jul 28 10:30:55.186090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186065    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container"
Jul 28 10:30:55.186090 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186071    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container"
Jul 28 10:30:55.186314 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186120    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kserve-container"
Jul 28 10:30:55.186314 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186128    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed33011d-7843-43e9-9b70-8db094db7646" containerName="kube-rbac-proxy"
Jul 28 10:30:55.186314 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186134    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kube-rbac-proxy"
Jul 28 10:30:55.186314 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.186139    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="98f5654d-0165-47e1-a1a1-ea27785211d5" containerName="kserve-container"
Jul 28 10:30:55.189343 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.189325    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.191625 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.191600    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-222a3-predictor-serving-cert\""
Jul 28 10:30:55.191747 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.191609    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-222a3-kube-rbac-proxy-sar-config\""
Jul 28 10:30:55.201240 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.201218    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"]
Jul 28 10:30:55.224459 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.224428    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.224609 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.224478    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df58082d-bb54-4891-9fde-dd207abea5a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.224609 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.224540    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bbx7\" (UniqueName: \"kubernetes.io/projected/df58082d-bb54-4891-9fde-dd207abea5a4-kube-api-access-8bbx7\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.296298 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.296268    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"]
Jul 28 10:30:55.299622 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.299602    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.301653 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.301632    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-222a3-predictor-serving-cert\""
Jul 28 10:30:55.301754 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.301639    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-222a3-kube-rbac-proxy-sar-config\""
Jul 28 10:30:55.311494 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.311468    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"]
Jul 28 10:30:55.324984 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.324961    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.325116 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.325000    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df58082d-bb54-4891-9fde-dd207abea5a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.325183 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:55.325132    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-222a3-predictor-serving-cert: secret "success-200-isvc-222a3-predictor-serving-cert" not found
Jul 28 10:30:55.325237 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.325196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6beb6ebc-7e48-4676-9361-992f85efd983-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.325349 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.325303    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bbx7\" (UniqueName: \"kubernetes.io/projected/df58082d-bb54-4891-9fde-dd207abea5a4-kube-api-access-8bbx7\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.325349 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:55.325323    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls podName:df58082d-bb54-4891-9fde-dd207abea5a4 nodeName:}" failed. No retries permitted until 2026-07-28 10:30:55.825299432 +0000 UTC m=+1363.400492343 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls") pod "success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" (UID: "df58082d-bb54-4891-9fde-dd207abea5a4") : secret "success-200-isvc-222a3-predictor-serving-cert" not found
Jul 28 10:30:55.325469 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.325361    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6q7vn\" (UniqueName: \"kubernetes.io/projected/6beb6ebc-7e48-4676-9361-992f85efd983-kube-api-access-6q7vn\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.325469 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.325402    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.325781 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.325763    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df58082d-bb54-4891-9fde-dd207abea5a4-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.335924 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.335902    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bbx7\" (UniqueName: \"kubernetes.io/projected/df58082d-bb54-4891-9fde-dd207abea5a4-kube-api-access-8bbx7\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.426675 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.426643    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6q7vn\" (UniqueName: \"kubernetes.io/projected/6beb6ebc-7e48-4676-9361-992f85efd983-kube-api-access-6q7vn\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.426887 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.426684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.426887 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:55.426814    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-222a3-predictor-serving-cert: secret "error-404-isvc-222a3-predictor-serving-cert" not found
Jul 28 10:30:55.426887 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:55.426861    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls podName:6beb6ebc-7e48-4676-9361-992f85efd983 nodeName:}" failed. No retries permitted until 2026-07-28 10:30:55.926847167 +0000 UTC m=+1363.502040075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls") pod "error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" (UID: "6beb6ebc-7e48-4676-9361-992f85efd983") : secret "error-404-isvc-222a3-predictor-serving-cert" not found
Jul 28 10:30:55.426887 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.426874    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6beb6ebc-7e48-4676-9361-992f85efd983-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.427469 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.427445    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6beb6ebc-7e48-4676-9361-992f85efd983-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.437105 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.437078    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6q7vn\" (UniqueName: \"kubernetes.io/projected/6beb6ebc-7e48-4676-9361-992f85efd983-kube-api-access-6q7vn\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.829533 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.829456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.831821 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.831799    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls\") pod \"success-200-isvc-222a3-predictor-6d5d87c54d-2tg26\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") " pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:55.930594 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.930562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:55.933030 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:55.933009    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls\") pod \"error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") " pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:56.069938 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:56.069901    2560 generic.go:356] "Generic (PLEG): container finished" podID="21380bc3-7d5d-4d24-987d-6013ac575626" containerID="9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac" exitCode=2
Jul 28 10:30:56.070112 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:56.069973    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" event={"ID":"21380bc3-7d5d-4d24-987d-6013ac575626","Type":"ContainerDied","Data":"9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac"}
Jul 28 10:30:56.071560 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:56.071540    2560 generic.go:356] "Generic (PLEG): container finished" podID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerID="e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76" exitCode=2
Jul 28 10:30:56.071657 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:56.071582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" event={"ID":"5493527c-3b35-4768-aa56-fbbcc9688dc3","Type":"ContainerDied","Data":"e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76"}
Jul 28 10:30:56.099245 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:56.099196    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:56.212489 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:56.212445    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:56.232168 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:56.232071    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"]
Jul 28 10:30:56.234582 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:30:56.234557    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf58082d_bb54_4891_9fde_dd207abea5a4.slice/crio-60d432bc6196862d39cd902cb4f7064b84485bc46f00a3599d6e000fabc2d68e WatchSource:0}: Error finding container 60d432bc6196862d39cd902cb4f7064b84485bc46f00a3599d6e000fabc2d68e: Status 404 returned error can't find the container with id 60d432bc6196862d39cd902cb4f7064b84485bc46f00a3599d6e000fabc2d68e
Jul 28 10:30:56.384539 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:56.384515    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"]
Jul 28 10:30:56.386599 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:30:56.386577    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6beb6ebc_7e48_4676_9361_992f85efd983.slice/crio-fd04bd4b88c0f6e3f819d996ec1039f1a29f743ce0c94a0772ee990a5f70be23 WatchSource:0}: Error finding container fd04bd4b88c0f6e3f819d996ec1039f1a29f743ce0c94a0772ee990a5f70be23: Status 404 returned error can't find the container with id fd04bd4b88c0f6e3f819d996ec1039f1a29f743ce0c94a0772ee990a5f70be23
Jul 28 10:30:57.077252 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.077211    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" event={"ID":"6beb6ebc-7e48-4676-9361-992f85efd983","Type":"ContainerStarted","Data":"be9c0a47b1f03f25451520b27842f992adf8de8b3f4e252efaab056f3c29e5a5"}
Jul 28 10:30:57.077457 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.077281    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" event={"ID":"6beb6ebc-7e48-4676-9361-992f85efd983","Type":"ContainerStarted","Data":"c9c9aaf6595ccc630583abd5b44c23351dd3c88913ced67d4d65ecd3feb0409f"}
Jul 28 10:30:57.077457 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.077312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" event={"ID":"6beb6ebc-7e48-4676-9361-992f85efd983","Type":"ContainerStarted","Data":"fd04bd4b88c0f6e3f819d996ec1039f1a29f743ce0c94a0772ee990a5f70be23"}
Jul 28 10:30:57.077457 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.077344    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:57.077457 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.077370    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:30:57.079055 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.079011    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:30:57.079188 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.079095    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" event={"ID":"df58082d-bb54-4891-9fde-dd207abea5a4","Type":"ContainerStarted","Data":"1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75"}
Jul 28 10:30:57.079188 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.079144    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" event={"ID":"df58082d-bb54-4891-9fde-dd207abea5a4","Type":"ContainerStarted","Data":"1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b"}
Jul 28 10:30:57.079188 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.079175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" event={"ID":"df58082d-bb54-4891-9fde-dd207abea5a4","Type":"ContainerStarted","Data":"60d432bc6196862d39cd902cb4f7064b84485bc46f00a3599d6e000fabc2d68e"}
Jul 28 10:30:57.079337 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.079211    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:57.079337 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.079242    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:30:57.080303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.080281    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:30:57.093771 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.093731    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podStartSLOduration=2.093700555 podStartE2EDuration="2.093700555s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:30:57.093077082 +0000 UTC m=+1364.668270014" watchObservedRunningTime="2026-07-28 10:30:57.093700555 +0000 UTC m=+1364.668893481"
Jul 28 10:30:57.110644 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.110590    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podStartSLOduration=2.110575921 podStartE2EDuration="2.110575921s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:30:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:30:57.108965349 +0000 UTC m=+1364.684158279" watchObservedRunningTime="2026-07-28 10:30:57.110575921 +0000 UTC m=+1364.685768850"
Jul 28 10:30:57.971491 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.971446    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.24:8643/healthz\": dial tcp 10.134.0.24:8643: connect: connection refused"
Jul 28 10:30:57.977367 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:57.977338    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.24:8080: connect: connection refused"
Jul 28 10:30:58.083192 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.083150    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:30:58.083411 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.083154    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:30:58.419349 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.419324    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:30:58.451848 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.451818    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-rbac-proxy-sar-config\") pod \"5493527c-3b35-4768-aa56-fbbcc9688dc3\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") "
Jul 28 10:30:58.452034 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.451896    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls\") pod \"5493527c-3b35-4768-aa56-fbbcc9688dc3\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") "
Jul 28 10:30:58.452034 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.451953    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pfxfq\" (UniqueName: \"kubernetes.io/projected/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-api-access-pfxfq\") pod \"5493527c-3b35-4768-aa56-fbbcc9688dc3\" (UID: \"5493527c-3b35-4768-aa56-fbbcc9688dc3\") "
Jul 28 10:30:58.452235 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.452213    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "5493527c-3b35-4768-aa56-fbbcc9688dc3" (UID: "5493527c-3b35-4768-aa56-fbbcc9688dc3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:30:58.453958 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.453927    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-api-access-pfxfq" (OuterVolumeSpecName: "kube-api-access-pfxfq") pod "5493527c-3b35-4768-aa56-fbbcc9688dc3" (UID: "5493527c-3b35-4768-aa56-fbbcc9688dc3"). InnerVolumeSpecName "kube-api-access-pfxfq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:30:58.454082 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.453932    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "5493527c-3b35-4768-aa56-fbbcc9688dc3" (UID: "5493527c-3b35-4768-aa56-fbbcc9688dc3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:30:58.553556 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.553479    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/5493527c-3b35-4768-aa56-fbbcc9688dc3-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:58.553556 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.553510    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfxfq\" (UniqueName: \"kubernetes.io/projected/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-api-access-pfxfq\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:58.553556 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.553520    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/5493527c-3b35-4768-aa56-fbbcc9688dc3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:58.944429 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:58.944400    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:30:59.058013 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.057979    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tbwz9\" (UniqueName: \"kubernetes.io/projected/21380bc3-7d5d-4d24-987d-6013ac575626-kube-api-access-tbwz9\") pod \"21380bc3-7d5d-4d24-987d-6013ac575626\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") "
Jul 28 10:30:59.058430 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.058027    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls\") pod \"21380bc3-7d5d-4d24-987d-6013ac575626\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") "
Jul 28 10:30:59.058430 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.058088    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21380bc3-7d5d-4d24-987d-6013ac575626-kube-rbac-proxy-sar-config\") pod \"21380bc3-7d5d-4d24-987d-6013ac575626\" (UID: \"21380bc3-7d5d-4d24-987d-6013ac575626\") "
Jul 28 10:30:59.058539 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.058468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21380bc3-7d5d-4d24-987d-6013ac575626-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "21380bc3-7d5d-4d24-987d-6013ac575626" (UID: "21380bc3-7d5d-4d24-987d-6013ac575626"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:30:59.059999 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.059978    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21380bc3-7d5d-4d24-987d-6013ac575626-kube-api-access-tbwz9" (OuterVolumeSpecName: "kube-api-access-tbwz9") pod "21380bc3-7d5d-4d24-987d-6013ac575626" (UID: "21380bc3-7d5d-4d24-987d-6013ac575626"). InnerVolumeSpecName "kube-api-access-tbwz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:30:59.060127 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.060107    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "21380bc3-7d5d-4d24-987d-6013ac575626" (UID: "21380bc3-7d5d-4d24-987d-6013ac575626"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:30:59.088839 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.088770    2560 generic.go:356] "Generic (PLEG): container finished" podID="21380bc3-7d5d-4d24-987d-6013ac575626" containerID="806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508" exitCode=0
Jul 28 10:30:59.088839 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.088821    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"
Jul 28 10:30:59.088839 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.088826    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" event={"ID":"21380bc3-7d5d-4d24-987d-6013ac575626","Type":"ContainerDied","Data":"806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508"}
Jul 28 10:30:59.089045 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.088880    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99" event={"ID":"21380bc3-7d5d-4d24-987d-6013ac575626","Type":"ContainerDied","Data":"bc1ec366be47389edab51639759b24326d859e477c4a53917f08e12ba10dd369"}
Jul 28 10:30:59.089045 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.088919    2560 scope.go:117] "RemoveContainer" containerID="9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac"
Jul 28 10:30:59.090575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.090547    2560 generic.go:356] "Generic (PLEG): container finished" podID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerID="be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4" exitCode=0
Jul 28 10:30:59.090682 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.090594    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" event={"ID":"5493527c-3b35-4768-aa56-fbbcc9688dc3","Type":"ContainerDied","Data":"be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4"}
Jul 28 10:30:59.090682 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.090599    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"
Jul 28 10:30:59.090682 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.090633    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5" event={"ID":"5493527c-3b35-4768-aa56-fbbcc9688dc3","Type":"ContainerDied","Data":"f69cac0a663ea4a54a1e5eb540bbdc2ae2951d53f5fbcd4069a321db6494c72a"}
Jul 28 10:30:59.099554 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.099541    2560 scope.go:117] "RemoveContainer" containerID="806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508"
Jul 28 10:30:59.107125 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.107101    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"]
Jul 28 10:30:59.107593 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.107580    2560 scope.go:117] "RemoveContainer" containerID="9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac"
Jul 28 10:30:59.107864 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:59.107844    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac\": container with ID starting with 9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac not found: ID does not exist" containerID="9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac"
Jul 28 10:30:59.107962 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.107869    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac"} err="failed to get container status \"9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac\": rpc error: code = NotFound desc = could not find container \"9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac\": container with ID starting with 9ddf0cc3c009a14b74a683ffbd043e3774355529407b1df4a33b459f7f9c1aac not found: ID does not exist"
Jul 28 10:30:59.107962 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.107885    2560 scope.go:117] "RemoveContainer" containerID="806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508"
Jul 28 10:30:59.108147 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:59.108129    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508\": container with ID starting with 806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508 not found: ID does not exist" containerID="806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508"
Jul 28 10:30:59.108189 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.108150    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508"} err="failed to get container status \"806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508\": rpc error: code = NotFound desc = could not find container \"806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508\": container with ID starting with 806f37822351a42b868483a872c671c943e2d236cb48c6ae86f6c6b781e57508 not found: ID does not exist"
Jul 28 10:30:59.108189 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.108163    2560 scope.go:117] "RemoveContainer" containerID="e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76"
Jul 28 10:30:59.111884 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.111863    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-0b9d4-predictor-7fdbfdf779-f5hq5"]
Jul 28 10:30:59.115636 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.115618    2560 scope.go:117] "RemoveContainer" containerID="be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4"
Jul 28 10:30:59.122920 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.122903    2560 scope.go:117] "RemoveContainer" containerID="e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76"
Jul 28 10:30:59.123168 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:59.123149    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76\": container with ID starting with e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76 not found: ID does not exist" containerID="e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76"
Jul 28 10:30:59.123230 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.123174    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76"} err="failed to get container status \"e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76\": rpc error: code = NotFound desc = could not find container \"e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76\": container with ID starting with e006dbd7acbe6bc9561a88162962ed0308d1d988defe551c7156df115ba19f76 not found: ID does not exist"
Jul 28 10:30:59.123230 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.123188    2560 scope.go:117] "RemoveContainer" containerID="be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4"
Jul 28 10:30:59.123230 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.123190    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"]
Jul 28 10:30:59.123393 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:30:59.123370    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4\": container with ID starting with be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4 not found: ID does not exist" containerID="be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4"
Jul 28 10:30:59.123432 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.123401    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4"} err="failed to get container status \"be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4\": rpc error: code = NotFound desc = could not find container \"be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4\": container with ID starting with be8d7a43fe3c1100460f606db9fb9c9034db9f24c17c774352a3e443b483c6a4 not found: ID does not exist"
Jul 28 10:30:59.125498 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.125479    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-0b9d4-predictor-54b67cb57d-48n99"]
Jul 28 10:30:59.159477 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.159450    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/21380bc3-7d5d-4d24-987d-6013ac575626-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:59.159477 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.159474    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbwz9\" (UniqueName: \"kubernetes.io/projected/21380bc3-7d5d-4d24-987d-6013ac575626-kube-api-access-tbwz9\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:30:59.159477 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:30:59.159484    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/21380bc3-7d5d-4d24-987d-6013ac575626-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:31:00.977898 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:00.977865    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" path="/var/lib/kubelet/pods/21380bc3-7d5d-4d24-987d-6013ac575626/volumes"
Jul 28 10:31:00.978293 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:00.978279    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" path="/var/lib/kubelet/pods/5493527c-3b35-4768-aa56-fbbcc9688dc3/volumes"
Jul 28 10:31:03.087458 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:03.087420    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:31:03.087950 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:03.087839    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:31:03.087950 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:03.087908    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:31:03.088309 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:03.088284    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:31:13.088663 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:13.088623    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:31:13.089041 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:13.088624    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:31:15.576099 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.576068    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"]
Jul 28 10:31:15.576570 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.576303    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container" containerID="cri-o://0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f" gracePeriod=30
Jul 28 10:31:15.576570 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.576356    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kube-rbac-proxy" containerID="cri-o://e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6" gracePeriod=30
Jul 28 10:31:15.633635 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.633604    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"]
Jul 28 10:31:15.634182 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.633835    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container" containerID="cri-o://3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557" gracePeriod=30
Jul 28 10:31:15.634182 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.633913    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kube-rbac-proxy" containerID="cri-o://49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c" gracePeriod=30
Jul 28 10:31:15.635957 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.635920    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"]
Jul 28 10:31:15.636431 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636402    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kube-rbac-proxy"
Jul 28 10:31:15.636431 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636426    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kube-rbac-proxy"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636471    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kube-rbac-proxy"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636480    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kube-rbac-proxy"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636491    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636500    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636511    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636519    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636589    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kube-rbac-proxy"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636602    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kserve-container"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636614    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="21380bc3-7d5d-4d24-987d-6013ac575626" containerName="kserve-container"
Jul 28 10:31:15.636669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.636623    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="5493527c-3b35-4768-aa56-fbbcc9688dc3" containerName="kube-rbac-proxy"
Jul 28 10:31:15.640956 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.640935    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.642906 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.642882    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4e070-kube-rbac-proxy-sar-config\""
Jul 28 10:31:15.643009 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.642910    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-4e070-predictor-serving-cert\""
Jul 28 10:31:15.649890 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.649865    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"]
Jul 28 10:31:15.746356 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.746322    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"]
Jul 28 10:31:15.749949 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.749927    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:15.752347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.752326    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4e070-predictor-serving-cert\""
Jul 28 10:31:15.752514 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.752359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-4e070-kube-rbac-proxy-sar-config\""
Jul 28 10:31:15.760741 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.760694    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"]
Jul 28 10:31:15.797704 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.797678    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tktl\" (UniqueName: \"kubernetes.io/projected/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-api-access-5tktl\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.797845 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.797707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.797845 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.797750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.899015 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.898980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tktl\" (UniqueName: \"kubernetes.io/projected/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-api-access-5tktl\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.899015 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.899021    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.899213 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.899048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.899213 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.899141    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qzlh\" (UniqueName: \"kubernetes.io/projected/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-api-access-6qzlh\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:15.899213 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:31:15.899152    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-4e070-predictor-serving-cert: secret "success-200-isvc-4e070-predictor-serving-cert" not found
Jul 28 10:31:15.899213 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.899171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:15.899213 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.899199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493e083a-503f-41a3-a0b7-b7e03faedef9-proxy-tls\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:15.899404 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:31:15.899219    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls podName:3116e4c0-2c47-4bab-8349-431399f7ac3b nodeName:}" failed. No retries permitted until 2026-07-28 10:31:16.399199439 +0000 UTC m=+1383.974392347 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls") pod "success-200-isvc-4e070-predictor-6985d68b68-kls5q" (UID: "3116e4c0-2c47-4bab-8349-431399f7ac3b") : secret "success-200-isvc-4e070-predictor-serving-cert" not found
Jul 28 10:31:15.899749 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.899706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.907572 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.907549    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tktl\" (UniqueName: \"kubernetes.io/projected/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-api-access-5tktl\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:15.999948 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.999914    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:15.999948 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:15.999950    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493e083a-503f-41a3-a0b7-b7e03faedef9-proxy-tls\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:16.000158 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.000027    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qzlh\" (UniqueName: \"kubernetes.io/projected/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-api-access-6qzlh\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:16.000575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.000557    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:16.002410 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.002385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493e083a-503f-41a3-a0b7-b7e03faedef9-proxy-tls\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:16.009173 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.009144    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qzlh\" (UniqueName: \"kubernetes.io/projected/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-api-access-6qzlh\") pod \"error-404-isvc-4e070-predictor-f797d89df-t5768\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") " pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:16.061104 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.061076    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:16.162251 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.162175    2560 generic.go:356] "Generic (PLEG): container finished" podID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerID="49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c" exitCode=2
Jul 28 10:31:16.162251 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.162222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" event={"ID":"a8416209-8c09-4f1c-a627-65de821b7f0c","Type":"ContainerDied","Data":"49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c"}
Jul 28 10:31:16.164445 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.164422    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerID="e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6" exitCode=2
Jul 28 10:31:16.164566 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.164463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" event={"ID":"e3ba1909-9221-445a-8ed6-4b277f54fcb4","Type":"ContainerDied","Data":"e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6"}
Jul 28 10:31:16.196087 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.196064    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"]
Jul 28 10:31:16.201006 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:31:16.200976    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod493e083a_503f_41a3_a0b7_b7e03faedef9.slice/crio-7e8f274e9caa66544ad233e75d37089322220e53e58d91145d762fa72a8ab126 WatchSource:0}: Error finding container 7e8f274e9caa66544ad233e75d37089322220e53e58d91145d762fa72a8ab126: Status 404 returned error can't find the container with id 7e8f274e9caa66544ad233e75d37089322220e53e58d91145d762fa72a8ab126
Jul 28 10:31:16.402955 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.402912    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:16.405246 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.405221    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls\") pod \"success-200-isvc-4e070-predictor-6985d68b68-kls5q\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") " pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:16.559288 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.559200    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:16.701647 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:16.701614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"]
Jul 28 10:31:16.703251 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:31:16.703227    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3116e4c0_2c47_4bab_8349_431399f7ac3b.slice/crio-c0780b153376111d1636fa336f9aa68168cf433ada50c4cab699bf71f0359ba3 WatchSource:0}: Error finding container c0780b153376111d1636fa336f9aa68168cf433ada50c4cab699bf71f0359ba3: Status 404 returned error can't find the container with id c0780b153376111d1636fa336f9aa68168cf433ada50c4cab699bf71f0359ba3
Jul 28 10:31:17.170443 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.170408    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" event={"ID":"3116e4c0-2c47-4bab-8349-431399f7ac3b","Type":"ContainerStarted","Data":"1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492"}
Jul 28 10:31:17.170649 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.170570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:17.170649 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.170630    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:17.170807 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.170659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" event={"ID":"3116e4c0-2c47-4bab-8349-431399f7ac3b","Type":"ContainerStarted","Data":"f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687"}
Jul 28 10:31:17.170807 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.170692    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" event={"ID":"3116e4c0-2c47-4bab-8349-431399f7ac3b","Type":"ContainerStarted","Data":"c0780b153376111d1636fa336f9aa68168cf433ada50c4cab699bf71f0359ba3"}
Jul 28 10:31:17.171972 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.171940    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:31:17.172339 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.172320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" event={"ID":"493e083a-503f-41a3-a0b7-b7e03faedef9","Type":"ContainerStarted","Data":"a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b"}
Jul 28 10:31:17.172399 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.172369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" event={"ID":"493e083a-503f-41a3-a0b7-b7e03faedef9","Type":"ContainerStarted","Data":"d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5"}
Jul 28 10:31:17.172399 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.172390    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" event={"ID":"493e083a-503f-41a3-a0b7-b7e03faedef9","Type":"ContainerStarted","Data":"7e8f274e9caa66544ad233e75d37089322220e53e58d91145d762fa72a8ab126"}
Jul 28 10:31:17.172576 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.172561    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:17.172637 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.172613    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:17.173438 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.173417    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 10:31:17.198873 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.198830    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podStartSLOduration=2.198816535 podStartE2EDuration="2.198816535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:31:17.197264112 +0000 UTC m=+1384.772457040" watchObservedRunningTime="2026-07-28 10:31:17.198816535 +0000 UTC m=+1384.774009464"
Jul 28 10:31:17.218842 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.218798    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podStartSLOduration=2.218785552 podStartE2EDuration="2.218785552s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:31:17.216944725 +0000 UTC m=+1384.792137655" watchObservedRunningTime="2026-07-28 10:31:17.218785552 +0000 UTC m=+1384.793978480"
Jul 28 10:31:17.868962 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.868918    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.27:8643/healthz\": dial tcp 10.134.0.27:8643: connect: connection refused"
Jul 28 10:31:17.869403 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:17.868917    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.26:8643/healthz\": dial tcp 10.134.0.26:8643: connect: connection refused"
Jul 28 10:31:18.176611 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:18.176570    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 10:31:18.176805 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:18.176640    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:31:19.673816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.673793    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:31:19.701049 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.700993    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:31:19.831919 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.831890    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6t5qf\" (UniqueName: \"kubernetes.io/projected/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-api-access-6t5qf\") pod \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") "
Jul 28 10:31:19.831919 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.831922    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8416209-8c09-4f1c-a627-65de821b7f0c-proxy-tls\") pod \"a8416209-8c09-4f1c-a627-65de821b7f0c\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") "
Jul 28 10:31:19.832135 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.831955    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f7l9l\" (UniqueName: \"kubernetes.io/projected/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-api-access-f7l9l\") pod \"a8416209-8c09-4f1c-a627-65de821b7f0c\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") "
Jul 28 10:31:19.832135 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.831987    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-rbac-proxy-sar-config\") pod \"a8416209-8c09-4f1c-a627-65de821b7f0c\" (UID: \"a8416209-8c09-4f1c-a627-65de821b7f0c\") "
Jul 28 10:31:19.832135 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.832029    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ba1909-9221-445a-8ed6-4b277f54fcb4-proxy-tls\") pod \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") "
Jul 28 10:31:19.832135 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.832052    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-rbac-proxy-sar-config\") pod \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\" (UID: \"e3ba1909-9221-445a-8ed6-4b277f54fcb4\") "
Jul 28 10:31:19.832454 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.832425    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8416209-8c09-4f1c-a627-65de821b7f0c" (UID: "a8416209-8c09-4f1c-a627-65de821b7f0c"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:31:19.832545 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.832444    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "e3ba1909-9221-445a-8ed6-4b277f54fcb4" (UID: "e3ba1909-9221-445a-8ed6-4b277f54fcb4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:31:19.834039 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.834017    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-api-access-6t5qf" (OuterVolumeSpecName: "kube-api-access-6t5qf") pod "e3ba1909-9221-445a-8ed6-4b277f54fcb4" (UID: "e3ba1909-9221-445a-8ed6-4b277f54fcb4"). InnerVolumeSpecName "kube-api-access-6t5qf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:31:19.834152 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.834057    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e3ba1909-9221-445a-8ed6-4b277f54fcb4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "e3ba1909-9221-445a-8ed6-4b277f54fcb4" (UID: "e3ba1909-9221-445a-8ed6-4b277f54fcb4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:31:19.834421 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.834405    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8416209-8c09-4f1c-a627-65de821b7f0c-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8416209-8c09-4f1c-a627-65de821b7f0c" (UID: "a8416209-8c09-4f1c-a627-65de821b7f0c"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:31:19.834463 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.834419    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-api-access-f7l9l" (OuterVolumeSpecName: "kube-api-access-f7l9l") pod "a8416209-8c09-4f1c-a627-65de821b7f0c" (UID: "a8416209-8c09-4f1c-a627-65de821b7f0c"). InnerVolumeSpecName "kube-api-access-f7l9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:31:19.932972 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.932935    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/e3ba1909-9221-445a-8ed6-4b277f54fcb4-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:31:19.932972 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.932966    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:31:19.933184 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.932977    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t5qf\" (UniqueName: \"kubernetes.io/projected/e3ba1909-9221-445a-8ed6-4b277f54fcb4-kube-api-access-6t5qf\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:31:19.933184 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.932997    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8416209-8c09-4f1c-a627-65de821b7f0c-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:31:19.933184 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.933006    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7l9l\" (UniqueName: \"kubernetes.io/projected/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-api-access-f7l9l\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:31:19.933184 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:19.933015    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8416209-8c09-4f1c-a627-65de821b7f0c-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:31:20.185235 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.185200    2560 generic.go:356] "Generic (PLEG): container finished" podID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerID="0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f" exitCode=0
Jul 28 10:31:20.185420 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.185257    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"
Jul 28 10:31:20.185420 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.185290    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" event={"ID":"e3ba1909-9221-445a-8ed6-4b277f54fcb4","Type":"ContainerDied","Data":"0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f"}
Jul 28 10:31:20.185420 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.185365    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w" event={"ID":"e3ba1909-9221-445a-8ed6-4b277f54fcb4","Type":"ContainerDied","Data":"a9146dad3855cc106bad6a95119801e0d3d117037d186b526e33841db5b1464a"}
Jul 28 10:31:20.185420 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.185395    2560 scope.go:117] "RemoveContainer" containerID="e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6"
Jul 28 10:31:20.187038 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.187016    2560 generic.go:356] "Generic (PLEG): container finished" podID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerID="3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557" exitCode=0
Jul 28 10:31:20.187169 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.187066    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"
Jul 28 10:31:20.187169 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.187096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" event={"ID":"a8416209-8c09-4f1c-a627-65de821b7f0c","Type":"ContainerDied","Data":"3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557"}
Jul 28 10:31:20.187169 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.187138    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp" event={"ID":"a8416209-8c09-4f1c-a627-65de821b7f0c","Type":"ContainerDied","Data":"097b45e649ed0cb14584ba5f662f3e36bcb70e91b083c6eb057e3636436c7af0"}
Jul 28 10:31:20.197918 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.197902    2560 scope.go:117] "RemoveContainer" containerID="0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f"
Jul 28 10:31:20.205152 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.205131    2560 scope.go:117] "RemoveContainer" containerID="e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6"
Jul 28 10:31:20.205395 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:31:20.205376    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6\": container with ID starting with e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6 not found: ID does not exist" containerID="e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6"
Jul 28 10:31:20.205467 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.205409    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6"} err="failed to get container status \"e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6\": rpc error: code = NotFound desc = could not find container \"e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6\": container with ID starting with e0f3b20dfb967d3e59b3511c672ca230466cbeef778b32f795a78e900ca965f6 not found: ID does not exist"
Jul 28 10:31:20.205467 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.205432    2560 scope.go:117] "RemoveContainer" containerID="0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f"
Jul 28 10:31:20.205695 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:31:20.205679    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f\": container with ID starting with 0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f not found: ID does not exist" containerID="0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f"
Jul 28 10:31:20.205787 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.205704    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f"} err="failed to get container status \"0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f\": rpc error: code = NotFound desc = could not find container \"0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f\": container with ID starting with 0cf20f8481dad1a5b81f95b9c49e39dbab39fb888c6166fbf1d894081b3a0d9f not found: ID does not exist"
Jul 28 10:31:20.205787 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.205739    2560 scope.go:117] "RemoveContainer" containerID="49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c"
Jul 28 10:31:20.211946 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.211926    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"]
Jul 28 10:31:20.212869 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.212854    2560 scope.go:117] "RemoveContainer" containerID="3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557"
Jul 28 10:31:20.216677 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.216657    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-af89d-predictor-6fc7bd7dcb-2989w"]
Jul 28 10:31:20.220215 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.220194    2560 scope.go:117] "RemoveContainer" containerID="49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c"
Jul 28 10:31:20.220467 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:31:20.220450    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c\": container with ID starting with 49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c not found: ID does not exist" containerID="49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c"
Jul 28 10:31:20.220529 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.220474    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c"} err="failed to get container status \"49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c\": rpc error: code = NotFound desc = could not find container \"49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c\": container with ID starting with 49079a8d1173acd30b8fbb123f5eccdfed28641536cc556f0f4f63357b13d59c not found: ID does not exist"
Jul 28 10:31:20.220529 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.220490    2560 scope.go:117] "RemoveContainer" containerID="3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557"
Jul 28 10:31:20.220771 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:31:20.220751    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557\": container with ID starting with 3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557 not found: ID does not exist" containerID="3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557"
Jul 28 10:31:20.220836 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.220780    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557"} err="failed to get container status \"3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557\": rpc error: code = NotFound desc = could not find container \"3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557\": container with ID starting with 3905d3f51dc4235b926d733669bd670179ddd041a305b37d84ec31d5a8c85557 not found: ID does not exist"
Jul 28 10:31:20.228492 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.228469    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"]
Jul 28 10:31:20.233403 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.233384    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-af89d-predictor-6bff89d85-j66sp"]
Jul 28 10:31:20.982910 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.982881    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" path="/var/lib/kubelet/pods/a8416209-8c09-4f1c-a627-65de821b7f0c/volumes"
Jul 28 10:31:20.983292 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:20.983279    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" path="/var/lib/kubelet/pods/e3ba1909-9221-445a-8ed6-4b277f54fcb4/volumes"
Jul 28 10:31:23.087885 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:23.087850    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:31:23.088316 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:23.088288    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:31:23.180766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:23.180743    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:31:23.181185 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:23.181166    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:31:23.181245 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:23.181228    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 10:31:23.181568 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:23.181543    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:31:33.087940 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:33.087893    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.29:8080: connect: connection refused"
Jul 28 10:31:33.088398 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:33.088289    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:31:33.181475 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:33.181440    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 10:31:33.181652 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:33.181563    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:31:43.088579 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:43.088547    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:31:43.089021 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:43.088803    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.28:8080: connect: connection refused"
Jul 28 10:31:43.181303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:43.181259    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 10:31:43.181558 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:43.181533    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:31:53.088856 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:53.088827    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:31:53.182109 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:53.182068    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 10:31:53.182275 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:31:53.182068    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:32:03.182325 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:03.182297    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:32:03.182822 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:03.182369    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:32:15.414099 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.414070    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"]
Jul 28 10:32:15.414525 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.414326    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container" containerID="cri-o://c9c9aaf6595ccc630583abd5b44c23351dd3c88913ced67d4d65ecd3feb0409f" gracePeriod=30
Jul 28 10:32:15.414525 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.414333    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kube-rbac-proxy" containerID="cri-o://be9c0a47b1f03f25451520b27842f992adf8de8b3f4e252efaab056f3c29e5a5" gracePeriod=30
Jul 28 10:32:15.470173 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.470139    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"]
Jul 28 10:32:15.470430 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.470405    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container" containerID="cri-o://1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b" gracePeriod=30
Jul 28 10:32:15.470551 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.470429    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kube-rbac-proxy" containerID="cri-o://1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75" gracePeriod=30
Jul 28 10:32:15.486889 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.486866    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"]
Jul 28 10:32:15.487175 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487163    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kube-rbac-proxy"
Jul 28 10:32:15.487216 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487176    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kube-rbac-proxy"
Jul 28 10:32:15.487216 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487199    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container"
Jul 28 10:32:15.487216 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487205    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container"
Jul 28 10:32:15.487216 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487213    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kube-rbac-proxy"
Jul 28 10:32:15.487338 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487219    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kube-rbac-proxy"
Jul 28 10:32:15.487338 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487227    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container"
Jul 28 10:32:15.487338 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487233    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container"
Jul 28 10:32:15.487338 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487291    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kserve-container"
Jul 28 10:32:15.487338 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487297    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kserve-container"
Jul 28 10:32:15.487338 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487302    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8416209-8c09-4f1c-a627-65de821b7f0c" containerName="kube-rbac-proxy"
Jul 28 10:32:15.487338 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.487309    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e3ba1909-9221-445a-8ed6-4b277f54fcb4" containerName="kube-rbac-proxy"
Jul 28 10:32:15.491625 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.491605    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.494323 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.494304    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8d409-kube-rbac-proxy-sar-config\""
Jul 28 10:32:15.494482 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.494462    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-8d409-predictor-serving-cert\""
Jul 28 10:32:15.510679 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.510651    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"]
Jul 28 10:32:15.555411 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.555376    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjj2x\" (UniqueName: \"kubernetes.io/projected/449aff97-4814-4b62-a575-b2095deefeaf-kube-api-access-hjj2x\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.555518 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.555426    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449aff97-4814-4b62-a575-b2095deefeaf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.555559 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.555516    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.591943 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.591914    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"]
Jul 28 10:32:15.595300 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.595282    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:15.598928 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.598901    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8d409-predictor-serving-cert\""
Jul 28 10:32:15.599034 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.598924    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-8d409-kube-rbac-proxy-sar-config\""
Jul 28 10:32:15.606864 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.606841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"]
Jul 28 10:32:15.656255 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.656222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjj2x\" (UniqueName: \"kubernetes.io/projected/449aff97-4814-4b62-a575-b2095deefeaf-kube-api-access-hjj2x\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.656487 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.656288    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449aff97-4814-4b62-a575-b2095deefeaf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.656487 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.656387    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.656487 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.656419    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8da85ee-464a-4e98-87cf-7175cc847352-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:15.656791 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:32:15.656583    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-8d409-predictor-serving-cert: secret "success-200-isvc-8d409-predictor-serving-cert" not found
Jul 28 10:32:15.656791 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:32:15.656644    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls podName:449aff97-4814-4b62-a575-b2095deefeaf nodeName:}" failed. No retries permitted until 2026-07-28 10:32:16.156624832 +0000 UTC m=+1443.731817742 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls") pod "success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" (UID: "449aff97-4814-4b62-a575-b2095deefeaf") : secret "success-200-isvc-8d409-predictor-serving-cert" not found
Jul 28 10:32:15.656791 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.656750    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:15.657035 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.656796    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq4cs\" (UniqueName: \"kubernetes.io/projected/a8da85ee-464a-4e98-87cf-7175cc847352-kube-api-access-fq4cs\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:15.657232 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.657210    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449aff97-4814-4b62-a575-b2095deefeaf-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.670664 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.670612    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjj2x\" (UniqueName: \"kubernetes.io/projected/449aff97-4814-4b62-a575-b2095deefeaf-kube-api-access-hjj2x\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:15.758046 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.758018    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8da85ee-464a-4e98-87cf-7175cc847352-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:15.758046 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.758050    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:15.758264 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.758071    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq4cs\" (UniqueName: \"kubernetes.io/projected/a8da85ee-464a-4e98-87cf-7175cc847352-kube-api-access-fq4cs\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:15.758264 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:32:15.758179    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-8d409-predictor-serving-cert: secret "error-404-isvc-8d409-predictor-serving-cert" not found
Jul 28 10:32:15.758264 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:32:15.758251    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls podName:a8da85ee-464a-4e98-87cf-7175cc847352 nodeName:}" failed. No retries permitted until 2026-07-28 10:32:16.258230661 +0000 UTC m=+1443.833423587 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls") pod "error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" (UID: "a8da85ee-464a-4e98-87cf-7175cc847352") : secret "error-404-isvc-8d409-predictor-serving-cert" not found
Jul 28 10:32:15.758645 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.758628    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8da85ee-464a-4e98-87cf-7175cc847352-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:15.772569 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:15.772547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq4cs\" (UniqueName: \"kubernetes.io/projected/a8da85ee-464a-4e98-87cf-7175cc847352-kube-api-access-fq4cs\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:16.162448 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.162416    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:16.164798 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.164775    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls\") pod \"success-200-isvc-8d409-predictor-5f6cb868f4-8njwq\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") " pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:16.263054 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.263024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:16.265203 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.265185    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls\") pod \"error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") " pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:16.401770 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.401742    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:16.403575 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.403549    2560 generic.go:356] "Generic (PLEG): container finished" podID="6beb6ebc-7e48-4676-9361-992f85efd983" containerID="be9c0a47b1f03f25451520b27842f992adf8de8b3f4e252efaab056f3c29e5a5" exitCode=2
Jul 28 10:32:16.403689 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.403621    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" event={"ID":"6beb6ebc-7e48-4676-9361-992f85efd983","Type":"ContainerDied","Data":"be9c0a47b1f03f25451520b27842f992adf8de8b3f4e252efaab056f3c29e5a5"}
Jul 28 10:32:16.405313 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.405294    2560 generic.go:356] "Generic (PLEG): container finished" podID="df58082d-bb54-4891-9fde-dd207abea5a4" containerID="1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75" exitCode=2
Jul 28 10:32:16.405387 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.405342    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" event={"ID":"df58082d-bb54-4891-9fde-dd207abea5a4","Type":"ContainerDied","Data":"1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75"}
Jul 28 10:32:16.505020 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.504996    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:16.538969 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.538939    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"]
Jul 28 10:32:16.546760 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:32:16.546732    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod449aff97_4814_4b62_a575_b2095deefeaf.slice/crio-7675dcea24b5db21600078c7f00ecace28be0573737063c5f49d996c3a8ee757 WatchSource:0}: Error finding container 7675dcea24b5db21600078c7f00ecace28be0573737063c5f49d996c3a8ee757: Status 404 returned error can't find the container with id 7675dcea24b5db21600078c7f00ecace28be0573737063c5f49d996c3a8ee757
Jul 28 10:32:16.647668 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:16.647639    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"]
Jul 28 10:32:16.657790 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:32:16.657762    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8da85ee_464a_4e98_87cf_7175cc847352.slice/crio-9098ce7374a730d4fce0aeb45c442fa204bd110eae9fb0bd8fbf02adacdc4dbc WatchSource:0}: Error finding container 9098ce7374a730d4fce0aeb45c442fa204bd110eae9fb0bd8fbf02adacdc4dbc: Status 404 returned error can't find the container with id 9098ce7374a730d4fce0aeb45c442fa204bd110eae9fb0bd8fbf02adacdc4dbc
Jul 28 10:32:17.411191 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.411155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" event={"ID":"449aff97-4814-4b62-a575-b2095deefeaf","Type":"ContainerStarted","Data":"6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643"}
Jul 28 10:32:17.411398 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.411229    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:17.411398 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.411260    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:17.411398 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.411282    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" event={"ID":"449aff97-4814-4b62-a575-b2095deefeaf","Type":"ContainerStarted","Data":"a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb"}
Jul 28 10:32:17.411398 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.411302    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" event={"ID":"449aff97-4814-4b62-a575-b2095deefeaf","Type":"ContainerStarted","Data":"7675dcea24b5db21600078c7f00ecace28be0573737063c5f49d996c3a8ee757"}
Jul 28 10:32:17.412766 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.412735    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:32:17.413274 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.413251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" event={"ID":"a8da85ee-464a-4e98-87cf-7175cc847352","Type":"ContainerStarted","Data":"a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078"}
Jul 28 10:32:17.413387 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.413295    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" event={"ID":"a8da85ee-464a-4e98-87cf-7175cc847352","Type":"ContainerStarted","Data":"48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b"}
Jul 28 10:32:17.413387 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.413320    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" event={"ID":"a8da85ee-464a-4e98-87cf-7175cc847352","Type":"ContainerStarted","Data":"9098ce7374a730d4fce0aeb45c442fa204bd110eae9fb0bd8fbf02adacdc4dbc"}
Jul 28 10:32:17.413387 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.413343    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:17.413387 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.413364    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:17.414366 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.414345    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:32:17.431664 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.431623    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podStartSLOduration=2.431610927 podStartE2EDuration="2.431610927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:32:17.430908702 +0000 UTC m=+1445.006101635" watchObservedRunningTime="2026-07-28 10:32:17.431610927 +0000 UTC m=+1445.006803855"
Jul 28 10:32:17.449606 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:17.449552    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podStartSLOduration=2.449536737 podStartE2EDuration="2.449536737s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:32:17.447888089 +0000 UTC m=+1445.023081031" watchObservedRunningTime="2026-07-28 10:32:17.449536737 +0000 UTC m=+1445.024729665"
Jul 28 10:32:18.084136 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.084095    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.28:8643/healthz\": dial tcp 10.134.0.28:8643: connect: connection refused"
Jul 28 10:32:18.084503 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.084101    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.29:8643/healthz\": dial tcp 10.134.0.29:8643: connect: connection refused"
Jul 28 10:32:18.419128 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.419094    2560 generic.go:356] "Generic (PLEG): container finished" podID="6beb6ebc-7e48-4676-9361-992f85efd983" containerID="c9c9aaf6595ccc630583abd5b44c23351dd3c88913ced67d4d65ecd3feb0409f" exitCode=0
Jul 28 10:32:18.419301 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.419157    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" event={"ID":"6beb6ebc-7e48-4676-9361-992f85efd983","Type":"ContainerDied","Data":"c9c9aaf6595ccc630583abd5b44c23351dd3c88913ced67d4d65ecd3feb0409f"}
Jul 28 10:32:18.419737 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.419702    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:32:18.419737 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.419702    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:32:18.512681 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.512655    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:32:18.580424 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.580400    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls\") pod \"6beb6ebc-7e48-4676-9361-992f85efd983\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") "
Jul 28 10:32:18.580559 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.580445    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6beb6ebc-7e48-4676-9361-992f85efd983-kube-rbac-proxy-sar-config\") pod \"6beb6ebc-7e48-4676-9361-992f85efd983\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") "
Jul 28 10:32:18.580559 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.580501    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6q7vn\" (UniqueName: \"kubernetes.io/projected/6beb6ebc-7e48-4676-9361-992f85efd983-kube-api-access-6q7vn\") pod \"6beb6ebc-7e48-4676-9361-992f85efd983\" (UID: \"6beb6ebc-7e48-4676-9361-992f85efd983\") "
Jul 28 10:32:18.580848 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.580825    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6beb6ebc-7e48-4676-9361-992f85efd983-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6beb6ebc-7e48-4676-9361-992f85efd983" (UID: "6beb6ebc-7e48-4676-9361-992f85efd983"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:32:18.582566 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.582540    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6beb6ebc-7e48-4676-9361-992f85efd983" (UID: "6beb6ebc-7e48-4676-9361-992f85efd983"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:32:18.582644 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.582592    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6beb6ebc-7e48-4676-9361-992f85efd983-kube-api-access-6q7vn" (OuterVolumeSpecName: "kube-api-access-6q7vn") pod "6beb6ebc-7e48-4676-9361-992f85efd983" (UID: "6beb6ebc-7e48-4676-9361-992f85efd983"). InnerVolumeSpecName "kube-api-access-6q7vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:32:18.613906 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.613885    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:32:18.681609 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.681551    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df58082d-bb54-4891-9fde-dd207abea5a4-kube-rbac-proxy-sar-config\") pod \"df58082d-bb54-4891-9fde-dd207abea5a4\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") "
Jul 28 10:32:18.681609 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.681581    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8bbx7\" (UniqueName: \"kubernetes.io/projected/df58082d-bb54-4891-9fde-dd207abea5a4-kube-api-access-8bbx7\") pod \"df58082d-bb54-4891-9fde-dd207abea5a4\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") "
Jul 28 10:32:18.681769 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.681622    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls\") pod \"df58082d-bb54-4891-9fde-dd207abea5a4\" (UID: \"df58082d-bb54-4891-9fde-dd207abea5a4\") "
Jul 28 10:32:18.681868 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.681850    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6beb6ebc-7e48-4676-9361-992f85efd983-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:32:18.681904 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.681877    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6beb6ebc-7e48-4676-9361-992f85efd983-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:32:18.681904 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.681891    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6q7vn\" (UniqueName: \"kubernetes.io/projected/6beb6ebc-7e48-4676-9361-992f85efd983-kube-api-access-6q7vn\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:32:18.681963 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.681896    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df58082d-bb54-4891-9fde-dd207abea5a4-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "df58082d-bb54-4891-9fde-dd207abea5a4" (UID: "df58082d-bb54-4891-9fde-dd207abea5a4"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:32:18.683546 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.683509    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "df58082d-bb54-4891-9fde-dd207abea5a4" (UID: "df58082d-bb54-4891-9fde-dd207abea5a4"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:32:18.683658 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.683620    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df58082d-bb54-4891-9fde-dd207abea5a4-kube-api-access-8bbx7" (OuterVolumeSpecName: "kube-api-access-8bbx7") pod "df58082d-bb54-4891-9fde-dd207abea5a4" (UID: "df58082d-bb54-4891-9fde-dd207abea5a4"). InnerVolumeSpecName "kube-api-access-8bbx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:32:18.783162 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.783134    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/df58082d-bb54-4891-9fde-dd207abea5a4-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:32:18.783162 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.783160    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/df58082d-bb54-4891-9fde-dd207abea5a4-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:32:18.783311 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:18.783174    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bbx7\" (UniqueName: \"kubernetes.io/projected/df58082d-bb54-4891-9fde-dd207abea5a4-kube-api-access-8bbx7\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:32:19.425006 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.424982    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"
Jul 28 10:32:19.425006 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.424989    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4" event={"ID":"6beb6ebc-7e48-4676-9361-992f85efd983","Type":"ContainerDied","Data":"fd04bd4b88c0f6e3f819d996ec1039f1a29f743ce0c94a0772ee990a5f70be23"}
Jul 28 10:32:19.425601 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.425051    2560 scope.go:117] "RemoveContainer" containerID="be9c0a47b1f03f25451520b27842f992adf8de8b3f4e252efaab056f3c29e5a5"
Jul 28 10:32:19.426591 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.426567    2560 generic.go:356] "Generic (PLEG): container finished" podID="df58082d-bb54-4891-9fde-dd207abea5a4" containerID="1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b" exitCode=0
Jul 28 10:32:19.426709 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.426611    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"
Jul 28 10:32:19.426709 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.426659    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" event={"ID":"df58082d-bb54-4891-9fde-dd207abea5a4","Type":"ContainerDied","Data":"1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b"}
Jul 28 10:32:19.426858 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.426726    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26" event={"ID":"df58082d-bb54-4891-9fde-dd207abea5a4","Type":"ContainerDied","Data":"60d432bc6196862d39cd902cb4f7064b84485bc46f00a3599d6e000fabc2d68e"}
Jul 28 10:32:19.433121 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.433106    2560 scope.go:117] "RemoveContainer" containerID="c9c9aaf6595ccc630583abd5b44c23351dd3c88913ced67d4d65ecd3feb0409f"
Jul 28 10:32:19.446260 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.446235    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"]
Jul 28 10:32:19.448495 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.448478    2560 scope.go:117] "RemoveContainer" containerID="1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75"
Jul 28 10:32:19.448656 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.448629    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-222a3-predictor-6d5d87c54d-2tg26"]
Jul 28 10:32:19.455692 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.455677    2560 scope.go:117] "RemoveContainer" containerID="1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b"
Jul 28 10:32:19.460137 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.460117    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"]
Jul 28 10:32:19.463458 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.463443    2560 scope.go:117] "RemoveContainer" containerID="1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75"
Jul 28 10:32:19.463749 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:32:19.463732    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75\": container with ID starting with 1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75 not found: ID does not exist" containerID="1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75"
Jul 28 10:32:19.463822 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.463756    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75"} err="failed to get container status \"1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75\": rpc error: code = NotFound desc = could not find container \"1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75\": container with ID starting with 1ef131d008020c0009b55afaec3c1cb63d3d8353147cfc5f255cdff32395ac75 not found: ID does not exist"
Jul 28 10:32:19.463822 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.463773    2560 scope.go:117] "RemoveContainer" containerID="1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b"
Jul 28 10:32:19.464025 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:32:19.464008    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b\": container with ID starting with 1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b not found: ID does not exist" containerID="1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b"
Jul 28 10:32:19.464065 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.464031    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b"} err="failed to get container status \"1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b\": rpc error: code = NotFound desc = could not find container \"1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b\": container with ID starting with 1a2fd1bd5fc048d360a5f8a0c20d9ce052f5260170330ca908f0c8972b0b029b not found: ID does not exist"
Jul 28 10:32:19.464462 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:19.464446    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-222a3-predictor-bb95bb5f7-jbnk4"]
Jul 28 10:32:20.977148 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:20.977118    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" path="/var/lib/kubelet/pods/6beb6ebc-7e48-4676-9361-992f85efd983/volumes"
Jul 28 10:32:20.977512 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:20.977500    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" path="/var/lib/kubelet/pods/df58082d-bb54-4891-9fde-dd207abea5a4/volumes"
Jul 28 10:32:23.423561 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:23.423536    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:32:23.424064 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:23.423923    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:32:23.424064 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:23.423969    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:32:23.424316 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:23.424293    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:32:33.424017 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:33.423982    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:32:33.424412 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:33.424392    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:32:43.424231 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:43.424194    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:32:43.424651 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:43.424337    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:32:53.423982 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:53.423936    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.32:8080: connect: connection refused"
Jul 28 10:32:53.424498 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:32:53.424375    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:33:03.424862 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:33:03.424834    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:33:03.425345 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:33:03.424912    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:33:12.914348 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:33:12.914317    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:33:12.915468 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:33:12.915450    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:38:12.939655 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:38:12.939611    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:38:12.946239 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:38:12.946208    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:40:30.511411 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.511359    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"]
Jul 28 10:40:30.513908 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.511683    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container" containerID="cri-o://f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687" gracePeriod=30
Jul 28 10:40:30.513908 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.511839    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kube-rbac-proxy" containerID="cri-o://1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492" gracePeriod=30
Jul 28 10:40:30.580895 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.580865    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"]
Jul 28 10:40:30.581255 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581239    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container"
Jul 28 10:40:30.581303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581256    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container"
Jul 28 10:40:30.581303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581269    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kube-rbac-proxy"
Jul 28 10:40:30.581303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581274    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kube-rbac-proxy"
Jul 28 10:40:30.581303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581286    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kube-rbac-proxy"
Jul 28 10:40:30.581303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581291    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kube-rbac-proxy"
Jul 28 10:40:30.581303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581298    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container"
Jul 28 10:40:30.581303 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581302    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container"
Jul 28 10:40:30.581540 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581357    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kube-rbac-proxy"
Jul 28 10:40:30.581540 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581365    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="6beb6ebc-7e48-4676-9361-992f85efd983" containerName="kserve-container"
Jul 28 10:40:30.581540 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581371    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kserve-container"
Jul 28 10:40:30.581540 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.581377    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="df58082d-bb54-4891-9fde-dd207abea5a4" containerName="kube-rbac-proxy"
Jul 28 10:40:30.584290 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.584274    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.586470 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.586439    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e803a-predictor-serving-cert\""
Jul 28 10:40:30.586680 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.586666    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-e803a-kube-rbac-proxy-sar-config\""
Jul 28 10:40:30.594009 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.593989    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"]
Jul 28 10:40:30.652283 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.652254    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt24d\" (UniqueName: \"kubernetes.io/projected/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-api-access-wt24d\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.652431 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.652333    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.652498 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.652470    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.669152 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.669118    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"]
Jul 28 10:40:30.669409 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.669365    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container" containerID="cri-o://d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5" gracePeriod=30
Jul 28 10:40:30.669504 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.669396    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kube-rbac-proxy" containerID="cri-o://a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b" gracePeriod=30
Jul 28 10:40:30.679987 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.679966    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"]
Jul 28 10:40:30.683333 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.683316    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:30.685900 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.685881    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e803a-predictor-serving-cert\""
Jul 28 10:40:30.685900 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.685894    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-e803a-kube-rbac-proxy-sar-config\""
Jul 28 10:40:30.692453 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.692433    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"]
Jul 28 10:40:30.753343 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.753315    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px5ds\" (UniqueName: \"kubernetes.io/projected/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-api-access-px5ds\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:30.753460 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.753350    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.753460 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.753369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt24d\" (UniqueName: \"kubernetes.io/projected/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-api-access-wt24d\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.753460 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.753418    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:30.753567 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:40:30.753473    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-e803a-predictor-serving-cert: secret "success-200-isvc-e803a-predictor-serving-cert" not found
Jul 28 10:40:30.753567 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.753473    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:30.753567 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:40:30.753533    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls podName:d66a12c5-150d-46dc-b847-11572a6b16c1 nodeName:}" failed. No retries permitted until 2026-07-28 10:40:31.253516926 +0000 UTC m=+1938.828709836 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls") pod "success-200-isvc-e803a-predictor-658895f866-4669k" (UID: "d66a12c5-150d-46dc-b847-11572a6b16c1") : secret "success-200-isvc-e803a-predictor-serving-cert" not found
Jul 28 10:40:30.753671 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.753585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.754209 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.754191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.762223 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.762170    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt24d\" (UniqueName: \"kubernetes.io/projected/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-api-access-wt24d\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:30.854342 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.854321    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px5ds\" (UniqueName: \"kubernetes.io/projected/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-api-access-px5ds\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:30.854453 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.854370    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:30.854453 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.854398    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:30.854565 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:40:30.854490    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-e803a-predictor-serving-cert: secret "error-404-isvc-e803a-predictor-serving-cert" not found
Jul 28 10:40:30.854565 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:40:30.854530    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls podName:a61fa5c3-b09f-4404-bac7-3cd38f5f0924 nodeName:}" failed. No retries permitted until 2026-07-28 10:40:31.354517085 +0000 UTC m=+1938.929709992 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls") pod "error-404-isvc-e803a-predictor-76bd58685f-h5jzr" (UID: "a61fa5c3-b09f-4404-bac7-3cd38f5f0924") : secret "error-404-isvc-e803a-predictor-serving-cert" not found
Jul 28 10:40:30.855214 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.855191    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:30.869186 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:30.869162    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-px5ds\" (UniqueName: \"kubernetes.io/projected/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-api-access-px5ds\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:31.255169 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.255135    2560 generic.go:356] "Generic (PLEG): container finished" podID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerID="a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b" exitCode=2
Jul 28 10:40:31.255340 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.255196    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" event={"ID":"493e083a-503f-41a3-a0b7-b7e03faedef9","Type":"ContainerDied","Data":"a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b"}
Jul 28 10:40:31.256885 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.256867    2560 generic.go:356] "Generic (PLEG): container finished" podID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerID="1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492" exitCode=2
Jul 28 10:40:31.257010 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.256901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" event={"ID":"3116e4c0-2c47-4bab-8349-431399f7ac3b","Type":"ContainerDied","Data":"1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492"}
Jul 28 10:40:31.257010 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.256937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:31.259221 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.259200    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls\") pod \"success-200-isvc-e803a-predictor-658895f866-4669k\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") " pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:31.357654 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.357629    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:31.360129 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.360108    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls\") pod \"error-404-isvc-e803a-predictor-76bd58685f-h5jzr\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") " pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:31.494723 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.494691    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:31.594048 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.594015    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:31.619393 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.619364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"]
Jul 28 10:40:31.621679 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:40:31.621654    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd66a12c5_150d_46dc_b847_11572a6b16c1.slice/crio-6a0d3fd5fc224168b71151865b1da6be49303d654f863f42c9386647d53ce6ca WatchSource:0}: Error finding container 6a0d3fd5fc224168b71151865b1da6be49303d654f863f42c9386647d53ce6ca: Status 404 returned error can't find the container with id 6a0d3fd5fc224168b71151865b1da6be49303d654f863f42c9386647d53ce6ca
Jul 28 10:40:31.736500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:31.736482    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"]
Jul 28 10:40:31.738181 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:40:31.738158    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61fa5c3_b09f_4404_bac7_3cd38f5f0924.slice/crio-1c682ca033f06aee78ef727f63daf42235b1d15db1d35261cd940148e02fb14c WatchSource:0}: Error finding container 1c682ca033f06aee78ef727f63daf42235b1d15db1d35261cd940148e02fb14c: Status 404 returned error can't find the container with id 1c682ca033f06aee78ef727f63daf42235b1d15db1d35261cd940148e02fb14c
Jul 28 10:40:32.262821 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.262787    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" event={"ID":"a61fa5c3-b09f-4404-bac7-3cd38f5f0924","Type":"ContainerStarted","Data":"c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5"}
Jul 28 10:40:32.263000 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.262858    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" event={"ID":"a61fa5c3-b09f-4404-bac7-3cd38f5f0924","Type":"ContainerStarted","Data":"22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72"}
Jul 28 10:40:32.263000 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.262881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" event={"ID":"a61fa5c3-b09f-4404-bac7-3cd38f5f0924","Type":"ContainerStarted","Data":"1c682ca033f06aee78ef727f63daf42235b1d15db1d35261cd940148e02fb14c"}
Jul 28 10:40:32.263000 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.262913    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:32.264436 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.264409    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" event={"ID":"d66a12c5-150d-46dc-b847-11572a6b16c1","Type":"ContainerStarted","Data":"4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce"}
Jul 28 10:40:32.264536 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.264468    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" event={"ID":"d66a12c5-150d-46dc-b847-11572a6b16c1","Type":"ContainerStarted","Data":"5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738"}
Jul 28 10:40:32.264536 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.264500    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" event={"ID":"d66a12c5-150d-46dc-b847-11572a6b16c1","Type":"ContainerStarted","Data":"6a0d3fd5fc224168b71151865b1da6be49303d654f863f42c9386647d53ce6ca"}
Jul 28 10:40:32.264665 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.264558    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:32.279852 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.279813    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podStartSLOduration=2.279799293 podStartE2EDuration="2.279799293s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:40:32.278829174 +0000 UTC m=+1939.854022104" watchObservedRunningTime="2026-07-28 10:40:32.279799293 +0000 UTC m=+1939.854992222"
Jul 28 10:40:32.295847 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:32.295809    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podStartSLOduration=2.29579668 podStartE2EDuration="2.29579668s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:40:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:40:32.294568085 +0000 UTC m=+1939.869761015" watchObservedRunningTime="2026-07-28 10:40:32.29579668 +0000 UTC m=+1939.870989608"
Jul 28 10:40:33.177190 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.177151    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.31:8643/healthz\": dial tcp 10.134.0.31:8643: connect: connection refused"
Jul 28 10:40:33.177560 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.177158    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.30:8643/healthz\": dial tcp 10.134.0.30:8643: connect: connection refused"
Jul 28 10:40:33.181428 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.181405    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.31:8080: connect: connection refused"
Jul 28 10:40:33.181588 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.181566    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.30:8080: connect: connection refused"
Jul 28 10:40:33.268405 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.268373    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:33.268584 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.268435    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:33.269268 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.269237    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:40:33.269383 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.269245    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:40:33.667078 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.667055    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:40:33.723557 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.723536    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:40:33.775930 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.775905    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-rbac-proxy-sar-config\") pod \"3116e4c0-2c47-4bab-8349-431399f7ac3b\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") "
Jul 28 10:40:33.776050 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.775941    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-rbac-proxy-sar-config\") pod \"493e083a-503f-41a3-a0b7-b7e03faedef9\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") "
Jul 28 10:40:33.776050 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.775998    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls\") pod \"3116e4c0-2c47-4bab-8349-431399f7ac3b\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") "
Jul 28 10:40:33.776050 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.776025    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tktl\" (UniqueName: \"kubernetes.io/projected/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-api-access-5tktl\") pod \"3116e4c0-2c47-4bab-8349-431399f7ac3b\" (UID: \"3116e4c0-2c47-4bab-8349-431399f7ac3b\") "
Jul 28 10:40:33.776466 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.776214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493e083a-503f-41a3-a0b7-b7e03faedef9-proxy-tls\") pod \"493e083a-503f-41a3-a0b7-b7e03faedef9\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") "
Jul 28 10:40:33.776466 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.776265    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qzlh\" (UniqueName: \"kubernetes.io/projected/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-api-access-6qzlh\") pod \"493e083a-503f-41a3-a0b7-b7e03faedef9\" (UID: \"493e083a-503f-41a3-a0b7-b7e03faedef9\") "
Jul 28 10:40:33.776466 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.776285    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3116e4c0-2c47-4bab-8349-431399f7ac3b" (UID: "3116e4c0-2c47-4bab-8349-431399f7ac3b"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:33.776466 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.776335    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "493e083a-503f-41a3-a0b7-b7e03faedef9" (UID: "493e083a-503f-41a3-a0b7-b7e03faedef9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:40:33.776687 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.776525    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:40:33.776687 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.776543    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:40:33.778060 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.778041    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-api-access-5tktl" (OuterVolumeSpecName: "kube-api-access-5tktl") pod "3116e4c0-2c47-4bab-8349-431399f7ac3b" (UID: "3116e4c0-2c47-4bab-8349-431399f7ac3b"). InnerVolumeSpecName "kube-api-access-5tktl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:33.778194 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.778174    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-api-access-6qzlh" (OuterVolumeSpecName: "kube-api-access-6qzlh") pod "493e083a-503f-41a3-a0b7-b7e03faedef9" (UID: "493e083a-503f-41a3-a0b7-b7e03faedef9"). InnerVolumeSpecName "kube-api-access-6qzlh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:40:33.778421 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.778396    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/493e083a-503f-41a3-a0b7-b7e03faedef9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "493e083a-503f-41a3-a0b7-b7e03faedef9" (UID: "493e083a-503f-41a3-a0b7-b7e03faedef9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:33.778421 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.778408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3116e4c0-2c47-4bab-8349-431399f7ac3b" (UID: "3116e4c0-2c47-4bab-8349-431399f7ac3b"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:40:33.877007 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.876973    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3116e4c0-2c47-4bab-8349-431399f7ac3b-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:40:33.877007 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.877004    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tktl\" (UniqueName: \"kubernetes.io/projected/3116e4c0-2c47-4bab-8349-431399f7ac3b-kube-api-access-5tktl\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:40:33.877007 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.877015    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/493e083a-503f-41a3-a0b7-b7e03faedef9-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:40:33.877186 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:33.877024    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qzlh\" (UniqueName: \"kubernetes.io/projected/493e083a-503f-41a3-a0b7-b7e03faedef9-kube-api-access-6qzlh\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:40:34.272853 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.272826    2560 generic.go:356] "Generic (PLEG): container finished" podID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerID="f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687" exitCode=0
Jul 28 10:40:34.273279 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.272867    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"
Jul 28 10:40:34.273279 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.272925    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" event={"ID":"3116e4c0-2c47-4bab-8349-431399f7ac3b","Type":"ContainerDied","Data":"f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687"}
Jul 28 10:40:34.273279 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.272984    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q" event={"ID":"3116e4c0-2c47-4bab-8349-431399f7ac3b","Type":"ContainerDied","Data":"c0780b153376111d1636fa336f9aa68168cf433ada50c4cab699bf71f0359ba3"}
Jul 28 10:40:34.273279 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.273011    2560 scope.go:117] "RemoveContainer" containerID="1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492"
Jul 28 10:40:34.274522 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.274496    2560 generic.go:356] "Generic (PLEG): container finished" podID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerID="d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5" exitCode=0
Jul 28 10:40:34.274618 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.274538    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"
Jul 28 10:40:34.274697 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.274606    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" event={"ID":"493e083a-503f-41a3-a0b7-b7e03faedef9","Type":"ContainerDied","Data":"d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5"}
Jul 28 10:40:34.274697 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.274669    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768" event={"ID":"493e083a-503f-41a3-a0b7-b7e03faedef9","Type":"ContainerDied","Data":"7e8f274e9caa66544ad233e75d37089322220e53e58d91145d762fa72a8ab126"}
Jul 28 10:40:34.275191 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.275157    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:40:34.275348 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.275319    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:40:34.281515 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.281500    2560 scope.go:117] "RemoveContainer" containerID="f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687"
Jul 28 10:40:34.289022 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.288995    2560 scope.go:117] "RemoveContainer" containerID="1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492"
Jul 28 10:40:34.289270 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:40:34.289252    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492\": container with ID starting with 1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492 not found: ID does not exist" containerID="1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492"
Jul 28 10:40:34.289326 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.289276    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492"} err="failed to get container status \"1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492\": rpc error: code = NotFound desc = could not find container \"1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492\": container with ID starting with 1488e6cf80416ddd466f9a5a2e477c0bf02c9d165f9689150d6fce52cc550492 not found: ID does not exist"
Jul 28 10:40:34.289326 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.289290    2560 scope.go:117] "RemoveContainer" containerID="f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687"
Jul 28 10:40:34.289513 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:40:34.289493    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687\": container with ID starting with f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687 not found: ID does not exist" containerID="f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687"
Jul 28 10:40:34.289582 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.289522    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687"} err="failed to get container status \"f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687\": rpc error: code = NotFound desc = could not find container \"f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687\": container with ID starting with f3a352995c429956b29eabdd60ca855bcc4f5b3423aca45a5032fbcef3ed8687 not found: ID does not exist"
Jul 28 10:40:34.289582 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.289544    2560 scope.go:117] "RemoveContainer" containerID="a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b"
Jul 28 10:40:34.296585 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.296565    2560 scope.go:117] "RemoveContainer" containerID="d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5"
Jul 28 10:40:34.298424 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.298379    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"]
Jul 28 10:40:34.302634 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.302614    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-4e070-predictor-6985d68b68-kls5q"]
Jul 28 10:40:34.306384 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.306365    2560 scope.go:117] "RemoveContainer" containerID="a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b"
Jul 28 10:40:34.306645 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:40:34.306627    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b\": container with ID starting with a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b not found: ID does not exist" containerID="a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b"
Jul 28 10:40:34.306758 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.306657    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b"} err="failed to get container status \"a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b\": rpc error: code = NotFound desc = could not find container \"a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b\": container with ID starting with a8e17ea49218d0c95c5e175f9a93ac228ab28cc13194d2cdd86eed3b3c46b82b not found: ID does not exist"
Jul 28 10:40:34.306758 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.306679    2560 scope.go:117] "RemoveContainer" containerID="d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5"
Jul 28 10:40:34.306952 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:40:34.306935    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5\": container with ID starting with d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5 not found: ID does not exist" containerID="d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5"
Jul 28 10:40:34.306995 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.306957    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5"} err="failed to get container status \"d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5\": rpc error: code = NotFound desc = could not find container \"d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5\": container with ID starting with d18c3c47c1cec0fd4db3c3cf161a438a77604a8b30b3134e02c81e40d7baf0d5 not found: ID does not exist"
Jul 28 10:40:34.314906 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.314884    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"]
Jul 28 10:40:34.319247 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.319228    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-4e070-predictor-f797d89df-t5768"]
Jul 28 10:40:34.977363 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.977331    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" path="/var/lib/kubelet/pods/3116e4c0-2c47-4bab-8349-431399f7ac3b/volumes"
Jul 28 10:40:34.977755 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:34.977740    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" path="/var/lib/kubelet/pods/493e083a-503f-41a3-a0b7-b7e03faedef9/volumes"
Jul 28 10:40:39.280666 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:39.280642    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:40:39.281051 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:39.280727    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:40:39.281115 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:39.281087    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:40:39.281343 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:39.281324    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:40:49.281305 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:49.281267    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:40:49.281774 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:49.281269    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:40:59.281166 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:59.281122    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:40:59.281561 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:40:59.281265    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:41:09.281966 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:09.281877    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.35:8080: connect: connection refused"
Jul 28 10:41:09.281966 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:09.281891    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.34:8080: connect: connection refused"
Jul 28 10:41:19.281854 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:19.281821    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:41:19.282269 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:19.282252    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:41:30.172505 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.172473    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"]
Jul 28 10:41:30.173025 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.172823    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container" containerID="cri-o://a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb" gracePeriod=30
Jul 28 10:41:30.173111 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.173057    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kube-rbac-proxy" containerID="cri-o://6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643" gracePeriod=30
Jul 28 10:41:30.224948 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.224914    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"]
Jul 28 10:41:30.225164 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.225137    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container" containerID="cri-o://48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b" gracePeriod=30
Jul 28 10:41:30.225283 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.225180    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kube-rbac-proxy" containerID="cri-o://a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078" gracePeriod=30
Jul 28 10:41:30.229520 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229498    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"]
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229843    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229857    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229865    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229871    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229880    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kube-rbac-proxy"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229885    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kube-rbac-proxy"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229909    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kube-rbac-proxy"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229916    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kube-rbac-proxy"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.229988    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kserve-container"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.230006    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kube-rbac-proxy"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.230017    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3116e4c0-2c47-4bab-8349-431399f7ac3b" containerName="kube-rbac-proxy"
Jul 28 10:41:30.230155 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.230025    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="493e083a-503f-41a3-a0b7-b7e03faedef9" containerName="kserve-container"
Jul 28 10:41:30.233337 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.233317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.235442 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.235423    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a647d-kube-rbac-proxy-sar-config\""
Jul 28 10:41:30.235535 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.235423    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a647d-predictor-serving-cert\""
Jul 28 10:41:30.244837 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.244813    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"]
Jul 28 10:41:30.303234 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.303205    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-proxy-tls\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.303413 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.303255    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.303413 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.303356    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8lcsb\" (UniqueName: \"kubernetes.io/projected/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-api-access-8lcsb\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.319991 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.319965    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"]
Jul 28 10:41:30.323206 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.323188    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.325433 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.325414    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a647d-predictor-serving-cert\""
Jul 28 10:41:30.325588 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.325572    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a647d-kube-rbac-proxy-sar-config\""
Jul 28 10:41:30.333974 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.333949    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"]
Jul 28 10:41:30.403903 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.403878    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.404044 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.403917    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-proxy-tls\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.404044 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.403969    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f9mr\" (UniqueName: \"kubernetes.io/projected/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-api-access-8f9mr\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.404044 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.404008    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.404193 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.404088    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8lcsb\" (UniqueName: \"kubernetes.io/projected/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-api-access-8lcsb\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.404193 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.404125    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.404626 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.404605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.406578 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.406556    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-proxy-tls\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.413244 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.413225    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8lcsb\" (UniqueName: \"kubernetes.io/projected/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-api-access-8lcsb\") pod \"success-200-isvc-a647d-predictor-5474579676-s27ps\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") " pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.490440 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.490374    2560 generic.go:356] "Generic (PLEG): container finished" podID="449aff97-4814-4b62-a575-b2095deefeaf" containerID="6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643" exitCode=2
Jul 28 10:41:30.490566 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.490446    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" event={"ID":"449aff97-4814-4b62-a575-b2095deefeaf","Type":"ContainerDied","Data":"6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643"}
Jul 28 10:41:30.492072 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.492048    2560 generic.go:356] "Generic (PLEG): container finished" podID="a8da85ee-464a-4e98-87cf-7175cc847352" containerID="a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078" exitCode=2
Jul 28 10:41:30.492195 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.492113    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" event={"ID":"a8da85ee-464a-4e98-87cf-7175cc847352","Type":"ContainerDied","Data":"a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078"}
Jul 28 10:41:30.504839 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.504813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f9mr\" (UniqueName: \"kubernetes.io/projected/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-api-access-8f9mr\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.504941 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.504886    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.505011 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.504940    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.505070 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:30.505026    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a647d-predictor-serving-cert: secret "error-404-isvc-a647d-predictor-serving-cert" not found
Jul 28 10:41:30.505122 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:30.505094    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls podName:475ccac3-d5d6-4509-9c8d-60f7801a8f2e nodeName:}" failed. No retries permitted until 2026-07-28 10:41:31.00507597 +0000 UTC m=+1998.580268889 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls") pod "error-404-isvc-a647d-predictor-784bd7c46f-s7brw" (UID: "475ccac3-d5d6-4509-9c8d-60f7801a8f2e") : secret "error-404-isvc-a647d-predictor-serving-cert" not found
Jul 28 10:41:30.505566 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.505544    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.513558 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.513535    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f9mr\" (UniqueName: \"kubernetes.io/projected/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-api-access-8f9mr\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:30.547410 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.547383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:30.679585 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:30.679560    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"]
Jul 28 10:41:30.682158 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:41:30.682130    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1194a9bf_bc05_43b9_9b6b_354a76f2eab9.slice/crio-e7be3319456b2384325676c544b745d65b7cf058aab3dd92c5f8aa0c0f478a1a WatchSource:0}: Error finding container e7be3319456b2384325676c544b745d65b7cf058aab3dd92c5f8aa0c0f478a1a: Status 404 returned error can't find the container with id e7be3319456b2384325676c544b745d65b7cf058aab3dd92c5f8aa0c0f478a1a
Jul 28 10:41:31.009680 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.009602    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:31.011966 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.011946    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls\") pod \"error-404-isvc-a647d-predictor-784bd7c46f-s7brw\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") " pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:31.234547 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.234516    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:31.359399 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.359261    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"]
Jul 28 10:41:31.361930 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:41:31.361902    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod475ccac3_d5d6_4509_9c8d_60f7801a8f2e.slice/crio-fe3da27c47eb8f127c95a3a2df348cda251114195e8ae97df22357bb540dc1b8 WatchSource:0}: Error finding container fe3da27c47eb8f127c95a3a2df348cda251114195e8ae97df22357bb540dc1b8: Status 404 returned error can't find the container with id fe3da27c47eb8f127c95a3a2df348cda251114195e8ae97df22357bb540dc1b8
Jul 28 10:41:31.497160 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.497129    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" event={"ID":"475ccac3-d5d6-4509-9c8d-60f7801a8f2e","Type":"ContainerStarted","Data":"1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874"}
Jul 28 10:41:31.497298 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.497209    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:31.497368 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.497326    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:31.497427 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.497401    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" event={"ID":"475ccac3-d5d6-4509-9c8d-60f7801a8f2e","Type":"ContainerStarted","Data":"271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0"}
Jul 28 10:41:31.497483 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.497442    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" event={"ID":"475ccac3-d5d6-4509-9c8d-60f7801a8f2e","Type":"ContainerStarted","Data":"fe3da27c47eb8f127c95a3a2df348cda251114195e8ae97df22357bb540dc1b8"}
Jul 28 10:41:31.498573 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.498550    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:41:31.499066 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.499040    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" event={"ID":"1194a9bf-bc05-43b9-9b6b-354a76f2eab9","Type":"ContainerStarted","Data":"1000dc24b2c9a4042aa42693800676144c17a6502ecc66ff1cfe48a343f22e1b"}
Jul 28 10:41:31.499170 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.499086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" event={"ID":"1194a9bf-bc05-43b9-9b6b-354a76f2eab9","Type":"ContainerStarted","Data":"d3097639720bde7fd6bb1361da2c993367601a83712cffe049ee81a7179dd490"}
Jul 28 10:41:31.499170 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.499111    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" event={"ID":"1194a9bf-bc05-43b9-9b6b-354a76f2eab9","Type":"ContainerStarted","Data":"e7be3319456b2384325676c544b745d65b7cf058aab3dd92c5f8aa0c0f478a1a"}
Jul 28 10:41:31.499252 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.499176    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:31.514450 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.514409    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podStartSLOduration=1.514396104 podStartE2EDuration="1.514396104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:41:31.51338129 +0000 UTC m=+1999.088574219" watchObservedRunningTime="2026-07-28 10:41:31.514396104 +0000 UTC m=+1999.089589033"
Jul 28 10:41:31.530248 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:31.530209    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podStartSLOduration=1.530195382 podStartE2EDuration="1.530195382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:41:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:41:31.529355703 +0000 UTC m=+1999.104548647" watchObservedRunningTime="2026-07-28 10:41:31.530195382 +0000 UTC m=+1999.105388310"
Jul 28 10:41:32.503437 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:32.503407    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:32.503839 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:32.503437    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:41:32.504600 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:32.504578    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:41:33.205635 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.205610    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:41:33.331960 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.331880    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hjj2x\" (UniqueName: \"kubernetes.io/projected/449aff97-4814-4b62-a575-b2095deefeaf-kube-api-access-hjj2x\") pod \"449aff97-4814-4b62-a575-b2095deefeaf\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") "
Jul 28 10:41:33.331960 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.331914    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449aff97-4814-4b62-a575-b2095deefeaf-kube-rbac-proxy-sar-config\") pod \"449aff97-4814-4b62-a575-b2095deefeaf\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") "
Jul 28 10:41:33.332178 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.331970    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls\") pod \"449aff97-4814-4b62-a575-b2095deefeaf\" (UID: \"449aff97-4814-4b62-a575-b2095deefeaf\") "
Jul 28 10:41:33.332349 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.332324    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/449aff97-4814-4b62-a575-b2095deefeaf-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "449aff97-4814-4b62-a575-b2095deefeaf" (UID: "449aff97-4814-4b62-a575-b2095deefeaf"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:41:33.333987 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.333958    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "449aff97-4814-4b62-a575-b2095deefeaf" (UID: "449aff97-4814-4b62-a575-b2095deefeaf"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:33.334096 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.333990    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/449aff97-4814-4b62-a575-b2095deefeaf-kube-api-access-hjj2x" (OuterVolumeSpecName: "kube-api-access-hjj2x") pod "449aff97-4814-4b62-a575-b2095deefeaf" (UID: "449aff97-4814-4b62-a575-b2095deefeaf"). InnerVolumeSpecName "kube-api-access-hjj2x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:41:33.419778 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.419745    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.33:8643/healthz\": dial tcp 10.134.0.33:8643: connect: connection refused"
Jul 28 10:41:33.424631 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.424607    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.33:8080: connect: connection refused"
Jul 28 10:41:33.432570 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.432553    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hjj2x\" (UniqueName: \"kubernetes.io/projected/449aff97-4814-4b62-a575-b2095deefeaf-kube-api-access-hjj2x\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:33.432628 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.432573    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/449aff97-4814-4b62-a575-b2095deefeaf-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:33.432628 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.432582    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/449aff97-4814-4b62-a575-b2095deefeaf-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:33.508161 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.508135    2560 generic.go:356] "Generic (PLEG): container finished" podID="449aff97-4814-4b62-a575-b2095deefeaf" containerID="a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb" exitCode=0
Jul 28 10:41:33.508493 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.508196    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"
Jul 28 10:41:33.508493 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.508221    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" event={"ID":"449aff97-4814-4b62-a575-b2095deefeaf","Type":"ContainerDied","Data":"a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb"}
Jul 28 10:41:33.508493 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.508304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq" event={"ID":"449aff97-4814-4b62-a575-b2095deefeaf","Type":"ContainerDied","Data":"7675dcea24b5db21600078c7f00ecace28be0573737063c5f49d996c3a8ee757"}
Jul 28 10:41:33.508493 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.508329    2560 scope.go:117] "RemoveContainer" containerID="6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643"
Jul 28 10:41:33.508808 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.508789    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:41:33.516419 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.516401    2560 scope.go:117] "RemoveContainer" containerID="a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb"
Jul 28 10:41:33.523316 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.523297    2560 scope.go:117] "RemoveContainer" containerID="6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643"
Jul 28 10:41:33.523568 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:33.523548    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643\": container with ID starting with 6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643 not found: ID does not exist" containerID="6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643"
Jul 28 10:41:33.523619 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.523577    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643"} err="failed to get container status \"6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643\": rpc error: code = NotFound desc = could not find container \"6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643\": container with ID starting with 6a0d8fc5bf510f9ef609ead6c2025e111148a657353c0c610ef5a55aa474b643 not found: ID does not exist"
Jul 28 10:41:33.523619 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.523592    2560 scope.go:117] "RemoveContainer" containerID="a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb"
Jul 28 10:41:33.523852 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:33.523832    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb\": container with ID starting with a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb not found: ID does not exist" containerID="a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb"
Jul 28 10:41:33.523905 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.523861    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb"} err="failed to get container status \"a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb\": rpc error: code = NotFound desc = could not find container \"a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb\": container with ID starting with a459a5a1f857f217f137a7e6eb2d16c0febaababa686a0c03a594c859c3fbfcb not found: ID does not exist"
Jul 28 10:41:33.531740 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.531699    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"]
Jul 28 10:41:33.536455 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:33.536436    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-8d409-predictor-5f6cb868f4-8njwq"]
Jul 28 10:41:34.977455 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:34.977426    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="449aff97-4814-4b62-a575-b2095deefeaf" path="/var/lib/kubelet/pods/449aff97-4814-4b62-a575-b2095deefeaf/volumes"
Jul 28 10:41:36.172793 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.172773    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:41:36.256476 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.256416    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8da85ee-464a-4e98-87cf-7175cc847352-kube-rbac-proxy-sar-config\") pod \"a8da85ee-464a-4e98-87cf-7175cc847352\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") "
Jul 28 10:41:36.256476 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.256451    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fq4cs\" (UniqueName: \"kubernetes.io/projected/a8da85ee-464a-4e98-87cf-7175cc847352-kube-api-access-fq4cs\") pod \"a8da85ee-464a-4e98-87cf-7175cc847352\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") "
Jul 28 10:41:36.256658 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.256499    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls\") pod \"a8da85ee-464a-4e98-87cf-7175cc847352\" (UID: \"a8da85ee-464a-4e98-87cf-7175cc847352\") "
Jul 28 10:41:36.256729 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.256686    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8da85ee-464a-4e98-87cf-7175cc847352-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a8da85ee-464a-4e98-87cf-7175cc847352" (UID: "a8da85ee-464a-4e98-87cf-7175cc847352"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:41:36.258459 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.258439    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a8da85ee-464a-4e98-87cf-7175cc847352" (UID: "a8da85ee-464a-4e98-87cf-7175cc847352"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:36.258591 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.258571    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8da85ee-464a-4e98-87cf-7175cc847352-kube-api-access-fq4cs" (OuterVolumeSpecName: "kube-api-access-fq4cs") pod "a8da85ee-464a-4e98-87cf-7175cc847352" (UID: "a8da85ee-464a-4e98-87cf-7175cc847352"). InnerVolumeSpecName "kube-api-access-fq4cs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:41:36.357488 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.357458    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a8da85ee-464a-4e98-87cf-7175cc847352-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:36.357488 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.357487    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a8da85ee-464a-4e98-87cf-7175cc847352-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:36.357632 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.357496    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fq4cs\" (UniqueName: \"kubernetes.io/projected/a8da85ee-464a-4e98-87cf-7175cc847352-kube-api-access-fq4cs\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:36.521877 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.521814    2560 generic.go:356] "Generic (PLEG): container finished" podID="a8da85ee-464a-4e98-87cf-7175cc847352" containerID="48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b" exitCode=0
Jul 28 10:41:36.521877 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.521855    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"
Jul 28 10:41:36.521877 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.521869    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" event={"ID":"a8da85ee-464a-4e98-87cf-7175cc847352","Type":"ContainerDied","Data":"48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b"}
Jul 28 10:41:36.522059 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.521928    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j" event={"ID":"a8da85ee-464a-4e98-87cf-7175cc847352","Type":"ContainerDied","Data":"9098ce7374a730d4fce0aeb45c442fa204bd110eae9fb0bd8fbf02adacdc4dbc"}
Jul 28 10:41:36.522059 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.521958    2560 scope.go:117] "RemoveContainer" containerID="a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078"
Jul 28 10:41:36.530985 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.530971    2560 scope.go:117] "RemoveContainer" containerID="48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b"
Jul 28 10:41:36.537828 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.537812    2560 scope.go:117] "RemoveContainer" containerID="a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078"
Jul 28 10:41:36.538068 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:36.538048    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078\": container with ID starting with a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078 not found: ID does not exist" containerID="a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078"
Jul 28 10:41:36.538116 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.538076    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078"} err="failed to get container status \"a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078\": rpc error: code = NotFound desc = could not find container \"a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078\": container with ID starting with a8937153204e352cd53f4858a5c909b91f8d823652d5703cb2311b0483731078 not found: ID does not exist"
Jul 28 10:41:36.538116 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.538092    2560 scope.go:117] "RemoveContainer" containerID="48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b"
Jul 28 10:41:36.538293 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:36.538277    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b\": container with ID starting with 48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b not found: ID does not exist" containerID="48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b"
Jul 28 10:41:36.538329 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.538298    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b"} err="failed to get container status \"48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b\": rpc error: code = NotFound desc = could not find container \"48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b\": container with ID starting with 48fa062ac73c1be4c0a9658be40c5390edcee7a6a81737c936a3c4472928b91b not found: ID does not exist"
Jul 28 10:41:36.544540 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.544519    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"]
Jul 28 10:41:36.549014 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.548994    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-8d409-predictor-5fcc67fb8c-l8z5j"]
Jul 28 10:41:36.977694 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:36.977663    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" path="/var/lib/kubelet/pods/a8da85ee-464a-4e98-87cf-7175cc847352/volumes"
Jul 28 10:41:37.507914 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:37.507887    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:41:37.508320 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:37.508259    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:41:38.512696 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:38.512670    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:41:38.513194 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:38.513160    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:41:40.775505 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.775344    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"]
Jul 28 10:41:40.775985 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.775688    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container" containerID="cri-o://5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738" gracePeriod=30
Jul 28 10:41:40.776132 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.776104    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kube-rbac-proxy" containerID="cri-o://4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce" gracePeriod=30
Jul 28 10:41:40.832948 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.832916    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"]
Jul 28 10:41:40.833414 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833397    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kube-rbac-proxy"
Jul 28 10:41:40.833469 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833416    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kube-rbac-proxy"
Jul 28 10:41:40.833469 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833441    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container"
Jul 28 10:41:40.833469 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833450    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container"
Jul 28 10:41:40.833469 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833462    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container"
Jul 28 10:41:40.833860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833473    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container"
Jul 28 10:41:40.833860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833485    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kube-rbac-proxy"
Jul 28 10:41:40.833860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833493    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kube-rbac-proxy"
Jul 28 10:41:40.833860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833557    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kserve-container"
Jul 28 10:41:40.833860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833567    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kserve-container"
Jul 28 10:41:40.833860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833578    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8da85ee-464a-4e98-87cf-7175cc847352" containerName="kube-rbac-proxy"
Jul 28 10:41:40.833860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.833588    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="449aff97-4814-4b62-a575-b2095deefeaf" containerName="kube-rbac-proxy"
Jul 28 10:41:40.841518 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.841493    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"]
Jul 28 10:41:40.841653 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.841630    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:40.841871 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.841826    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container" containerID="cri-o://22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72" gracePeriod=30
Jul 28 10:41:40.842019 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.841999    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kube-rbac-proxy" containerID="cri-o://c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5" gracePeriod=30
Jul 28 10:41:40.847188 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.845968    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7711-kube-rbac-proxy-sar-config\""
Jul 28 10:41:40.847188 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.846200    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-f7711-predictor-serving-cert\""
Jul 28 10:41:40.847793 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.847618    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"]
Jul 28 10:41:40.894649 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.894617    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsfhn\" (UniqueName: \"kubernetes.io/projected/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-api-access-lsfhn\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:40.894649 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.894652    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:40.894889 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.894804    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:40.934599 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.934566    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"]
Jul 28 10:41:40.937997 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.937980    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:40.940200 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.940008    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f7711-predictor-serving-cert\""
Jul 28 10:41:40.940200 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.940087    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-f7711-kube-rbac-proxy-sar-config\""
Jul 28 10:41:40.948639 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.948548    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"]
Jul 28 10:41:40.995176 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.995151    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kkshh\" (UniqueName: \"kubernetes.io/projected/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-api-access-kkshh\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:40.995347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.995192    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:40.995347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.995233    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:40.995347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.995256    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:40.995347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.995280    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsfhn\" (UniqueName: \"kubernetes.io/projected/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-api-access-lsfhn\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:40.995347 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.995301    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:40.995347 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:40.995331    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-f7711-predictor-serving-cert: secret "success-200-isvc-f7711-predictor-serving-cert" not found
Jul 28 10:41:40.995630 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:40.995402    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls podName:ea213dbf-7200-4c6d-8f12-78e23917b867 nodeName:}" failed. No retries permitted until 2026-07-28 10:41:41.495374012 +0000 UTC m=+2009.070566930 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls") pod "success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" (UID: "ea213dbf-7200-4c6d-8f12-78e23917b867") : secret "success-200-isvc-f7711-predictor-serving-cert" not found
Jul 28 10:41:40.995911 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:40.995891    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:41.003844 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.003820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsfhn\" (UniqueName: \"kubernetes.io/projected/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-api-access-lsfhn\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:41.096423 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.096342    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:41.096423 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.096404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kkshh\" (UniqueName: \"kubernetes.io/projected/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-api-access-kkshh\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:41.096645 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.096429    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:41.096645 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:41.096543    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f7711-predictor-serving-cert: secret "error-404-isvc-f7711-predictor-serving-cert" not found
Jul 28 10:41:41.096645 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:41.096600    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls podName:8c320d1a-ea74-423e-b59b-21aaff97675f nodeName:}" failed. No retries permitted until 2026-07-28 10:41:41.596585938 +0000 UTC m=+2009.171778858 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls") pod "error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" (UID: "8c320d1a-ea74-423e-b59b-21aaff97675f") : secret "error-404-isvc-f7711-predictor-serving-cert" not found
Jul 28 10:41:41.097094 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.097070    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:41.108251 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.108219    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kkshh\" (UniqueName: \"kubernetes.io/projected/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-api-access-kkshh\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:41.500427 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.500395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:41.502687 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.502657    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls\") pod \"success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") " pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:41.547184 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.547154    2560 generic.go:356] "Generic (PLEG): container finished" podID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerID="c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5" exitCode=2
Jul 28 10:41:41.547414 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.547232    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" event={"ID":"a61fa5c3-b09f-4404-bac7-3cd38f5f0924","Type":"ContainerDied","Data":"c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5"}
Jul 28 10:41:41.549118 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.549089    2560 generic.go:356] "Generic (PLEG): container finished" podID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerID="4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce" exitCode=2
Jul 28 10:41:41.549235 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.549124    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" event={"ID":"d66a12c5-150d-46dc-b847-11572a6b16c1","Type":"ContainerDied","Data":"4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce"}
Jul 28 10:41:41.601530 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.601499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:41.601665 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:41.601649    2560 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-f7711-predictor-serving-cert: secret "error-404-isvc-f7711-predictor-serving-cert" not found
Jul 28 10:41:41.601756 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:41.601745    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls podName:8c320d1a-ea74-423e-b59b-21aaff97675f nodeName:}" failed. No retries permitted until 2026-07-28 10:41:42.601704409 +0000 UTC m=+2010.176897338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls") pod "error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" (UID: "8c320d1a-ea74-423e-b59b-21aaff97675f") : secret "error-404-isvc-f7711-predictor-serving-cert" not found
Jul 28 10:41:41.761424 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.761340    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:41.895371 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:41.895116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"]
Jul 28 10:41:41.897794 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:41:41.897770    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea213dbf_7200_4c6d_8f12_78e23917b867.slice/crio-2213992c038a65c23301fb5585d035f7ecb55af1f33d281677f916ed21678d29 WatchSource:0}: Error finding container 2213992c038a65c23301fb5585d035f7ecb55af1f33d281677f916ed21678d29: Status 404 returned error can't find the container with id 2213992c038a65c23301fb5585d035f7ecb55af1f33d281677f916ed21678d29
Jul 28 10:41:42.554771 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.554735    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" event={"ID":"ea213dbf-7200-4c6d-8f12-78e23917b867","Type":"ContainerStarted","Data":"f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0"}
Jul 28 10:41:42.554954 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.554803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" event={"ID":"ea213dbf-7200-4c6d-8f12-78e23917b867","Type":"ContainerStarted","Data":"0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4"}
Jul 28 10:41:42.554954 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.554833    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" event={"ID":"ea213dbf-7200-4c6d-8f12-78e23917b867","Type":"ContainerStarted","Data":"2213992c038a65c23301fb5585d035f7ecb55af1f33d281677f916ed21678d29"}
Jul 28 10:41:42.554954 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.554868    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:42.575663 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.575619    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podStartSLOduration=2.575603239 podStartE2EDuration="2.575603239s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:41:42.573847815 +0000 UTC m=+2010.149040744" watchObservedRunningTime="2026-07-28 10:41:42.575603239 +0000 UTC m=+2010.150796168"
Jul 28 10:41:42.610029 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.609998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:42.612399 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.612377    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls\") pod \"error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") " pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:42.752806 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.752768    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:42.888189 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:42.888154    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"]
Jul 28 10:41:42.892040 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:41:42.892010    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c320d1a_ea74_423e_b59b_21aaff97675f.slice/crio-de2ed9a0b4981ac80a2b1e7b3cca2163d9c6395a198e92e1c2fc7b73e7bc8f0a WatchSource:0}: Error finding container de2ed9a0b4981ac80a2b1e7b3cca2163d9c6395a198e92e1c2fc7b73e7bc8f0a: Status 404 returned error can't find the container with id de2ed9a0b4981ac80a2b1e7b3cca2163d9c6395a198e92e1c2fc7b73e7bc8f0a
Jul 28 10:41:43.560349 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:43.560310    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" event={"ID":"8c320d1a-ea74-423e-b59b-21aaff97675f","Type":"ContainerStarted","Data":"e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484"}
Jul 28 10:41:43.560851 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:43.560373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" event={"ID":"8c320d1a-ea74-423e-b59b-21aaff97675f","Type":"ContainerStarted","Data":"63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893"}
Jul 28 10:41:43.560851 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:43.560395    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" event={"ID":"8c320d1a-ea74-423e-b59b-21aaff97675f","Type":"ContainerStarted","Data":"de2ed9a0b4981ac80a2b1e7b3cca2163d9c6395a198e92e1c2fc7b73e7bc8f0a"}
Jul 28 10:41:43.560851 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:43.560665    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:43.562026 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:43.562002    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:41:43.577376 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:43.577332    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podStartSLOduration=3.57732052 podStartE2EDuration="3.57732052s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:41:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:41:43.576984491 +0000 UTC m=+2011.152177421" watchObservedRunningTime="2026-07-28 10:41:43.57732052 +0000 UTC m=+2011.152513449"
Jul 28 10:41:44.275880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.275836    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.34:8643/healthz\": dial tcp 10.134.0.34:8643: connect: connection refused"
Jul 28 10:41:44.276058 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.275836    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.35:8643/healthz\": dial tcp 10.134.0.35:8643: connect: connection refused"
Jul 28 10:41:44.528038 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.527982    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:41:44.567463 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.567429    2560 generic.go:356] "Generic (PLEG): container finished" podID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerID="5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738" exitCode=0
Jul 28 10:41:44.567880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.567477    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"
Jul 28 10:41:44.567880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.567510    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" event={"ID":"d66a12c5-150d-46dc-b847-11572a6b16c1","Type":"ContainerDied","Data":"5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738"}
Jul 28 10:41:44.567880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.567600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k" event={"ID":"d66a12c5-150d-46dc-b847-11572a6b16c1","Type":"ContainerDied","Data":"6a0d3fd5fc224168b71151865b1da6be49303d654f863f42c9386647d53ce6ca"}
Jul 28 10:41:44.567880 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.567635    2560 scope.go:117] "RemoveContainer" containerID="4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce"
Jul 28 10:41:44.568110 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.568033    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:41:44.568222 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.568206    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:44.568296 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.568272    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:44.569006 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.568985    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:44.575597 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.575582    2560 scope.go:117] "RemoveContainer" containerID="5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738"
Jul 28 10:41:44.582541 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.582524    2560 scope.go:117] "RemoveContainer" containerID="4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce"
Jul 28 10:41:44.582797 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:44.582777    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce\": container with ID starting with 4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce not found: ID does not exist" containerID="4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce"
Jul 28 10:41:44.582869 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.582806    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce"} err="failed to get container status \"4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce\": rpc error: code = NotFound desc = could not find container \"4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce\": container with ID starting with 4b3032f28fb3ac9c6420f0390bf8a91d8af3eae80f497922c161af9c1e1c26ce not found: ID does not exist"
Jul 28 10:41:44.582869 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.582822    2560 scope.go:117] "RemoveContainer" containerID="5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738"
Jul 28 10:41:44.583065 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:44.583044    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738\": container with ID starting with 5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738 not found: ID does not exist" containerID="5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738"
Jul 28 10:41:44.583158 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.583070    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738"} err="failed to get container status \"5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738\": rpc error: code = NotFound desc = could not find container \"5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738\": container with ID starting with 5e78bd966541fb8213f7bded90422b3e3b2ba083c9c4f2fd41b5177910638738 not found: ID does not exist"
Jul 28 10:41:44.625370 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.625345    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-rbac-proxy-sar-config\") pod \"d66a12c5-150d-46dc-b847-11572a6b16c1\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") "
Jul 28 10:41:44.625502 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.625404    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt24d\" (UniqueName: \"kubernetes.io/projected/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-api-access-wt24d\") pod \"d66a12c5-150d-46dc-b847-11572a6b16c1\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") "
Jul 28 10:41:44.625502 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.625426    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls\") pod \"d66a12c5-150d-46dc-b847-11572a6b16c1\" (UID: \"d66a12c5-150d-46dc-b847-11572a6b16c1\") "
Jul 28 10:41:44.625800 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.625701    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d66a12c5-150d-46dc-b847-11572a6b16c1" (UID: "d66a12c5-150d-46dc-b847-11572a6b16c1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:41:44.627519 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.627491    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d66a12c5-150d-46dc-b847-11572a6b16c1" (UID: "d66a12c5-150d-46dc-b847-11572a6b16c1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:44.627607 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.627569    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-api-access-wt24d" (OuterVolumeSpecName: "kube-api-access-wt24d") pod "d66a12c5-150d-46dc-b847-11572a6b16c1" (UID: "d66a12c5-150d-46dc-b847-11572a6b16c1"). InnerVolumeSpecName "kube-api-access-wt24d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:41:44.726458 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.726420    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:44.726458 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.726449    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt24d\" (UniqueName: \"kubernetes.io/projected/d66a12c5-150d-46dc-b847-11572a6b16c1-kube-api-access-wt24d\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:44.726458 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.726459    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d66a12c5-150d-46dc-b847-11572a6b16c1-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:44.888663 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.888601    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:41:44.889675 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.889653    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"]
Jul 28 10:41:44.894473 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.894450    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-e803a-predictor-658895f866-4669k"]
Jul 28 10:41:44.978899 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:44.978867    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" path="/var/lib/kubelet/pods/d66a12c5-150d-46dc-b847-11572a6b16c1/volumes"
Jul 28 10:41:45.029362 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.029324    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls\") pod \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") "
Jul 28 10:41:45.029530 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.029406    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-rbac-proxy-sar-config\") pod \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") "
Jul 28 10:41:45.029530 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.029440    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-px5ds\" (UniqueName: \"kubernetes.io/projected/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-api-access-px5ds\") pod \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\" (UID: \"a61fa5c3-b09f-4404-bac7-3cd38f5f0924\") "
Jul 28 10:41:45.029758 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.029703    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "a61fa5c3-b09f-4404-bac7-3cd38f5f0924" (UID: "a61fa5c3-b09f-4404-bac7-3cd38f5f0924"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:41:45.031465 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.031442    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-api-access-px5ds" (OuterVolumeSpecName: "kube-api-access-px5ds") pod "a61fa5c3-b09f-4404-bac7-3cd38f5f0924" (UID: "a61fa5c3-b09f-4404-bac7-3cd38f5f0924"). InnerVolumeSpecName "kube-api-access-px5ds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:41:45.031551 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.031473    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "a61fa5c3-b09f-4404-bac7-3cd38f5f0924" (UID: "a61fa5c3-b09f-4404-bac7-3cd38f5f0924"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:41:45.130556 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.130526    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:45.130556 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.130557    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px5ds\" (UniqueName: \"kubernetes.io/projected/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-kube-api-access-px5ds\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:45.130788 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.130571    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/a61fa5c3-b09f-4404-bac7-3cd38f5f0924-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:41:45.574219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.574139    2560 generic.go:356] "Generic (PLEG): container finished" podID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerID="22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72" exitCode=0
Jul 28 10:41:45.574219 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.574190    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"
Jul 28 10:41:45.574661 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.574222    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" event={"ID":"a61fa5c3-b09f-4404-bac7-3cd38f5f0924","Type":"ContainerDied","Data":"22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72"}
Jul 28 10:41:45.574661 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.574283    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr" event={"ID":"a61fa5c3-b09f-4404-bac7-3cd38f5f0924","Type":"ContainerDied","Data":"1c682ca033f06aee78ef727f63daf42235b1d15db1d35261cd940148e02fb14c"}
Jul 28 10:41:45.574661 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.574314    2560 scope.go:117] "RemoveContainer" containerID="c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5"
Jul 28 10:41:45.575563 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.575482    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:45.582581 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.582567    2560 scope.go:117] "RemoveContainer" containerID="22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72"
Jul 28 10:41:45.589631 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.589616    2560 scope.go:117] "RemoveContainer" containerID="c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5"
Jul 28 10:41:45.589938 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:45.589916    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5\": container with ID starting with c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5 not found: ID does not exist" containerID="c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5"
Jul 28 10:41:45.590001 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.589946    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5"} err="failed to get container status \"c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5\": rpc error: code = NotFound desc = could not find container \"c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5\": container with ID starting with c2c13e16d50446235037516571ae7928bd79fa0a1b9c74a2492e1539764a26a5 not found: ID does not exist"
Jul 28 10:41:45.590001 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.589961    2560 scope.go:117] "RemoveContainer" containerID="22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72"
Jul 28 10:41:45.590214 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:41:45.590190    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72\": container with ID starting with 22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72 not found: ID does not exist" containerID="22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72"
Jul 28 10:41:45.590337 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.590217    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72"} err="failed to get container status \"22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72\": rpc error: code = NotFound desc = could not find container \"22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72\": container with ID starting with 22e88f3cac85bbc86f3f43a0e6e80ccc00f10ca4bd4fd7f48857ccf0ad628b72 not found: ID does not exist"
Jul 28 10:41:45.596403 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.596382    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"]
Jul 28 10:41:45.600638 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:45.600618    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-e803a-predictor-76bd58685f-h5jzr"]
Jul 28 10:41:46.977695 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:46.977663    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" path="/var/lib/kubelet/pods/a61fa5c3-b09f-4404-bac7-3cd38f5f0924/volumes"
Jul 28 10:41:47.509339 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:47.509301    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:41:48.513159 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:48.513122    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:41:49.573131 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:49.573101    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:41:49.573669 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:49.573644    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:41:50.581083 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:50.581055    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:41:50.581552 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:50.581524    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:41:57.508893 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:57.508852    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:41:58.513885 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:58.513846    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:41:59.573643 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:41:59.573606    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:42:00.581500 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:00.581466    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:42:07.508996 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:07.508951    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.37:8080: connect: connection refused"
Jul 28 10:42:08.513216 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:08.513175    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.36:8080: connect: connection refused"
Jul 28 10:42:09.574273 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:09.574230    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:42:10.581967 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:10.581928    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:42:17.509395 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:17.509361    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:42:18.514367 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:18.514338    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:42:19.574393 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:19.574358    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.38:8080: connect: connection refused"
Jul 28 10:42:20.582004 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:20.581970    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 28 10:42:29.574703 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:29.574673    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:42:30.582254 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:42:30.582228    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:43:12.961808 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:43:12.961778    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:43:12.969000 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:43:12.968975    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:48:12.985121 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:48:12.985097    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:48:12.992574 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:48:12.992549    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:50:55.741538 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:55.741497    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"]
Jul 28 10:50:55.744039 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:55.741740    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container" containerID="cri-o://0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4" gracePeriod=30
Jul 28 10:50:55.744039 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:55.741791    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kube-rbac-proxy" containerID="cri-o://f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0" gracePeriod=30
Jul 28 10:50:55.816548 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:55.816507    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"]
Jul 28 10:50:55.816801 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:55.816777    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container" containerID="cri-o://63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893" gracePeriod=30
Jul 28 10:50:55.816929 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:55.816874    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kube-rbac-proxy" containerID="cri-o://e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484" gracePeriod=30
Jul 28 10:50:56.657094 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:56.657060    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerID="e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484" exitCode=2
Jul 28 10:50:56.657292 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:56.657131    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" event={"ID":"8c320d1a-ea74-423e-b59b-21aaff97675f","Type":"ContainerDied","Data":"e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484"}
Jul 28 10:50:56.658876 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:56.658855    2560 generic.go:356] "Generic (PLEG): container finished" podID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerID="f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0" exitCode=2
Jul 28 10:50:56.659000 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:56.658930    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" event={"ID":"ea213dbf-7200-4c6d-8f12-78e23917b867","Type":"ContainerDied","Data":"f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0"}
Jul 28 10:50:59.038584 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.038564    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:50:59.146371 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.146339    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lsfhn\" (UniqueName: \"kubernetes.io/projected/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-api-access-lsfhn\") pod \"ea213dbf-7200-4c6d-8f12-78e23917b867\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") "
Jul 28 10:50:59.146567 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.146406    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-rbac-proxy-sar-config\") pod \"ea213dbf-7200-4c6d-8f12-78e23917b867\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") "
Jul 28 10:50:59.146567 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.146436    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls\") pod \"ea213dbf-7200-4c6d-8f12-78e23917b867\" (UID: \"ea213dbf-7200-4c6d-8f12-78e23917b867\") "
Jul 28 10:50:59.146872 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.146843    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "ea213dbf-7200-4c6d-8f12-78e23917b867" (UID: "ea213dbf-7200-4c6d-8f12-78e23917b867"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:50:59.148544 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.148522    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "ea213dbf-7200-4c6d-8f12-78e23917b867" (UID: "ea213dbf-7200-4c6d-8f12-78e23917b867"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:50:59.148621 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.148573    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-api-access-lsfhn" (OuterVolumeSpecName: "kube-api-access-lsfhn") pod "ea213dbf-7200-4c6d-8f12-78e23917b867" (UID: "ea213dbf-7200-4c6d-8f12-78e23917b867"). InnerVolumeSpecName "kube-api-access-lsfhn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:50:59.162207 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.162187    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:50:59.247260 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.247226    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls\") pod \"8c320d1a-ea74-423e-b59b-21aaff97675f\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") "
Jul 28 10:50:59.247260 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.247259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-rbac-proxy-sar-config\") pod \"8c320d1a-ea74-423e-b59b-21aaff97675f\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") "
Jul 28 10:50:59.247499 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.247282    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kkshh\" (UniqueName: \"kubernetes.io/projected/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-api-access-kkshh\") pod \"8c320d1a-ea74-423e-b59b-21aaff97675f\" (UID: \"8c320d1a-ea74-423e-b59b-21aaff97675f\") "
Jul 28 10:50:59.247499 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.247442    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsfhn\" (UniqueName: \"kubernetes.io/projected/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-api-access-lsfhn\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:50:59.247499 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.247462    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/ea213dbf-7200-4c6d-8f12-78e23917b867-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:50:59.247499 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.247476    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/ea213dbf-7200-4c6d-8f12-78e23917b867-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:50:59.247648 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.247593    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8c320d1a-ea74-423e-b59b-21aaff97675f" (UID: "8c320d1a-ea74-423e-b59b-21aaff97675f"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:50:59.249315 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.249296    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-api-access-kkshh" (OuterVolumeSpecName: "kube-api-access-kkshh") pod "8c320d1a-ea74-423e-b59b-21aaff97675f" (UID: "8c320d1a-ea74-423e-b59b-21aaff97675f"). InnerVolumeSpecName "kube-api-access-kkshh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:50:59.249360 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.249321    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8c320d1a-ea74-423e-b59b-21aaff97675f" (UID: "8c320d1a-ea74-423e-b59b-21aaff97675f"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:50:59.347981 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.347946    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kkshh\" (UniqueName: \"kubernetes.io/projected/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-api-access-kkshh\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:50:59.347981 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.347982    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8c320d1a-ea74-423e-b59b-21aaff97675f-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:50:59.347981 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.347998    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8c320d1a-ea74-423e-b59b-21aaff97675f-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:50:59.672345 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.672311    2560 generic.go:356] "Generic (PLEG): container finished" podID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerID="0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4" exitCode=0
Jul 28 10:50:59.672529 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.672368    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"
Jul 28 10:50:59.672529 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.672373    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" event={"ID":"ea213dbf-7200-4c6d-8f12-78e23917b867","Type":"ContainerDied","Data":"0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4"}
Jul 28 10:50:59.672529 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.672461    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv" event={"ID":"ea213dbf-7200-4c6d-8f12-78e23917b867","Type":"ContainerDied","Data":"2213992c038a65c23301fb5585d035f7ecb55af1f33d281677f916ed21678d29"}
Jul 28 10:50:59.672529 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.672496    2560 scope.go:117] "RemoveContainer" containerID="f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0"
Jul 28 10:50:59.674202 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.674176    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerID="63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893" exitCode=0
Jul 28 10:50:59.674301 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.674233    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" event={"ID":"8c320d1a-ea74-423e-b59b-21aaff97675f","Type":"ContainerDied","Data":"63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893"}
Jul 28 10:50:59.674301 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.674260    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"
Jul 28 10:50:59.674301 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.674280    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk" event={"ID":"8c320d1a-ea74-423e-b59b-21aaff97675f","Type":"ContainerDied","Data":"de2ed9a0b4981ac80a2b1e7b3cca2163d9c6395a198e92e1c2fc7b73e7bc8f0a"}
Jul 28 10:50:59.682113 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.682095    2560 scope.go:117] "RemoveContainer" containerID="0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4"
Jul 28 10:50:59.689236 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.689219    2560 scope.go:117] "RemoveContainer" containerID="f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0"
Jul 28 10:50:59.689473 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:50:59.689454    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0\": container with ID starting with f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0 not found: ID does not exist" containerID="f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0"
Jul 28 10:50:59.689522 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.689482    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0"} err="failed to get container status \"f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0\": rpc error: code = NotFound desc = could not find container \"f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0\": container with ID starting with f5d9a4e8fd31b6e0c06b8fc50ad020a2237e829ce5077f78d3ab8f5b504b8bf0 not found: ID does not exist"
Jul 28 10:50:59.689522 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.689497    2560 scope.go:117] "RemoveContainer" containerID="0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4"
Jul 28 10:50:59.689725 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:50:59.689698    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4\": container with ID starting with 0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4 not found: ID does not exist" containerID="0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4"
Jul 28 10:50:59.689773 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.689740    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4"} err="failed to get container status \"0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4\": rpc error: code = NotFound desc = could not find container \"0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4\": container with ID starting with 0110894d83263f11a497af7dd3b11fa00bed079f468781943ca0722a105d67a4 not found: ID does not exist"
Jul 28 10:50:59.689773 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.689760    2560 scope.go:117] "RemoveContainer" containerID="e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484"
Jul 28 10:50:59.696928 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.696837    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"]
Jul 28 10:50:59.696984 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.696957    2560 scope.go:117] "RemoveContainer" containerID="63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893"
Jul 28 10:50:59.703972 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.703950    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-f7711-predictor-7b6f54fcd8-dkzjv"]
Jul 28 10:50:59.704526 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.704513    2560 scope.go:117] "RemoveContainer" containerID="e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484"
Jul 28 10:50:59.704794 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:50:59.704775    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484\": container with ID starting with e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484 not found: ID does not exist" containerID="e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484"
Jul 28 10:50:59.704885 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.704804    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484"} err="failed to get container status \"e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484\": rpc error: code = NotFound desc = could not find container \"e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484\": container with ID starting with e586cc2005f39cb1bbbbde65541e1179da94a64e1c50c44f24bbd6c94b265484 not found: ID does not exist"
Jul 28 10:50:59.704885 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.704828    2560 scope.go:117] "RemoveContainer" containerID="63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893"
Jul 28 10:50:59.705069 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:50:59.705049    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893\": container with ID starting with 63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893 not found: ID does not exist" containerID="63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893"
Jul 28 10:50:59.705140 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.705078    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893"} err="failed to get container status \"63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893\": rpc error: code = NotFound desc = could not find container \"63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893\": container with ID starting with 63b8fcd45e2920ec06098bea09311fa48490103eb56f11b8a1ccfb7a48501893 not found: ID does not exist"
Jul 28 10:50:59.715247 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.715227    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"]
Jul 28 10:50:59.719795 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:50:59.719771    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-f7711-predictor-6d447c4c9c-vwjvk"]
Jul 28 10:51:00.983338 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:51:00.983300    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" path="/var/lib/kubelet/pods/8c320d1a-ea74-423e-b59b-21aaff97675f/volumes"
Jul 28 10:51:00.983939 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:51:00.983918    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" path="/var/lib/kubelet/pods/ea213dbf-7200-4c6d-8f12-78e23917b867/volumes"
Jul 28 10:53:13.005618 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:53:13.005549    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:53:13.014041 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:53:13.014017    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:58:13.026149 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:13.026084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:58:13.035664 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:13.035643    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:58:49.554209 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:49.554176    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"]
Jul 28 10:58:49.554838 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:49.554438    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container" containerID="cri-o://d3097639720bde7fd6bb1361da2c993367601a83712cffe049ee81a7179dd490" gracePeriod=30
Jul 28 10:58:49.554838 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:49.554468    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kube-rbac-proxy" containerID="cri-o://1000dc24b2c9a4042aa42693800676144c17a6502ecc66ff1cfe48a343f22e1b" gracePeriod=30
Jul 28 10:58:49.610141 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:49.610102    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"]
Jul 28 10:58:49.610377 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:49.610352    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container" containerID="cri-o://271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0" gracePeriod=30
Jul 28 10:58:49.610468 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:49.610439    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kube-rbac-proxy" containerID="cri-o://1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874" gracePeriod=30
Jul 28 10:58:50.357794 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:50.357762    2560 generic.go:356] "Generic (PLEG): container finished" podID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerID="1000dc24b2c9a4042aa42693800676144c17a6502ecc66ff1cfe48a343f22e1b" exitCode=2
Jul 28 10:58:50.357989 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:50.357843    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" event={"ID":"1194a9bf-bc05-43b9-9b6b-354a76f2eab9","Type":"ContainerDied","Data":"1000dc24b2c9a4042aa42693800676144c17a6502ecc66ff1cfe48a343f22e1b"}
Jul 28 10:58:50.359337 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:50.359317    2560 generic.go:356] "Generic (PLEG): container finished" podID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerID="1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874" exitCode=2
Jul 28 10:58:50.359463 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:50.359360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" event={"ID":"475ccac3-d5d6-4509-9c8d-60f7801a8f2e","Type":"ContainerDied","Data":"1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874"}
Jul 28 10:58:52.369635 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.369603    2560 generic.go:356] "Generic (PLEG): container finished" podID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerID="d3097639720bde7fd6bb1361da2c993367601a83712cffe049ee81a7179dd490" exitCode=0
Jul 28 10:58:52.370027 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.369662    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" event={"ID":"1194a9bf-bc05-43b9-9b6b-354a76f2eab9","Type":"ContainerDied","Data":"d3097639720bde7fd6bb1361da2c993367601a83712cffe049ee81a7179dd490"}
Jul 28 10:58:52.402014 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.401993    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:58:52.504101 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.504010    2560 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.37:8643/healthz\": dial tcp 10.134.0.37:8643: connect: connection refused"
Jul 28 10:58:52.510623 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.510602    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8lcsb\" (UniqueName: \"kubernetes.io/projected/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-api-access-8lcsb\") pod \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") "
Jul 28 10:58:52.510710 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.510663    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-proxy-tls\") pod \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") "
Jul 28 10:58:52.510710 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.510694    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-rbac-proxy-sar-config\") pod \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\" (UID: \"1194a9bf-bc05-43b9-9b6b-354a76f2eab9\") "
Jul 28 10:58:52.511084 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.511059    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "1194a9bf-bc05-43b9-9b6b-354a76f2eab9" (UID: "1194a9bf-bc05-43b9-9b6b-354a76f2eab9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:58:52.512728 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.512692    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-api-access-8lcsb" (OuterVolumeSpecName: "kube-api-access-8lcsb") pod "1194a9bf-bc05-43b9-9b6b-354a76f2eab9" (UID: "1194a9bf-bc05-43b9-9b6b-354a76f2eab9"). InnerVolumeSpecName "kube-api-access-8lcsb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:58:52.512826 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.512725    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "1194a9bf-bc05-43b9-9b6b-354a76f2eab9" (UID: "1194a9bf-bc05-43b9-9b6b-354a76f2eab9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:58:52.611923 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.611895    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:58:52.611923 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.611918    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:58:52.611923 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.611929    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8lcsb\" (UniqueName: \"kubernetes.io/projected/1194a9bf-bc05-43b9-9b6b-354a76f2eab9-kube-api-access-8lcsb\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:58:52.745773 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.745748    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:58:52.813881 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.813797    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls\") pod \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") "
Jul 28 10:58:52.813881 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.813842    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8f9mr\" (UniqueName: \"kubernetes.io/projected/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-api-access-8f9mr\") pod \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") "
Jul 28 10:58:52.813881 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.813861    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-rbac-proxy-sar-config\") pod \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\" (UID: \"475ccac3-d5d6-4509-9c8d-60f7801a8f2e\") "
Jul 28 10:58:52.814229 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.814205    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "475ccac3-d5d6-4509-9c8d-60f7801a8f2e" (UID: "475ccac3-d5d6-4509-9c8d-60f7801a8f2e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 28 10:58:52.815904 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.815880    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "475ccac3-d5d6-4509-9c8d-60f7801a8f2e" (UID: "475ccac3-d5d6-4509-9c8d-60f7801a8f2e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 28 10:58:52.815983 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.815948    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-api-access-8f9mr" (OuterVolumeSpecName: "kube-api-access-8f9mr") pod "475ccac3-d5d6-4509-9c8d-60f7801a8f2e" (UID: "475ccac3-d5d6-4509-9c8d-60f7801a8f2e"). InnerVolumeSpecName "kube-api-access-8f9mr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 28 10:58:52.914892 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.914858    2560 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-proxy-tls\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:58:52.914892 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.914885    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8f9mr\" (UniqueName: \"kubernetes.io/projected/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-api-access-8f9mr\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:58:52.914892 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:52.914898    2560 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/475ccac3-d5d6-4509-9c8d-60f7801a8f2e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-131-51.ec2.internal\" DevicePath \"\""
Jul 28 10:58:53.375169 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.375135    2560 generic.go:356] "Generic (PLEG): container finished" podID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerID="271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0" exitCode=0
Jul 28 10:58:53.375553 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.375178    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"
Jul 28 10:58:53.375553 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.375223    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" event={"ID":"475ccac3-d5d6-4509-9c8d-60f7801a8f2e","Type":"ContainerDied","Data":"271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0"}
Jul 28 10:58:53.375553 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.375283    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw" event={"ID":"475ccac3-d5d6-4509-9c8d-60f7801a8f2e","Type":"ContainerDied","Data":"fe3da27c47eb8f127c95a3a2df348cda251114195e8ae97df22357bb540dc1b8"}
Jul 28 10:58:53.375553 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.375315    2560 scope.go:117] "RemoveContainer" containerID="1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874"
Jul 28 10:58:53.377023 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.377001    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"
Jul 28 10:58:53.377142 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.376992    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps" event={"ID":"1194a9bf-bc05-43b9-9b6b-354a76f2eab9","Type":"ContainerDied","Data":"e7be3319456b2384325676c544b745d65b7cf058aab3dd92c5f8aa0c0f478a1a"}
Jul 28 10:58:53.383437 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.383418    2560 scope.go:117] "RemoveContainer" containerID="271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0"
Jul 28 10:58:53.390209 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.390193    2560 scope.go:117] "RemoveContainer" containerID="1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874"
Jul 28 10:58:53.390468 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:58:53.390451    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874\": container with ID starting with 1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874 not found: ID does not exist" containerID="1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874"
Jul 28 10:58:53.390513 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.390477    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874"} err="failed to get container status \"1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874\": rpc error: code = NotFound desc = could not find container \"1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874\": container with ID starting with 1bb1bd5a3f077817e82352d3f0387d20e0524882366b28f903e07460da602874 not found: ID does not exist"
Jul 28 10:58:53.390513 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.390493    2560 scope.go:117] "RemoveContainer" containerID="271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0"
Jul 28 10:58:53.390782 ip-10-0-131-51 kubenswrapper[2560]: E0728 10:58:53.390726    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0\": container with ID starting with 271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0 not found: ID does not exist" containerID="271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0"
Jul 28 10:58:53.390782 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.390753    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0"} err="failed to get container status \"271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0\": rpc error: code = NotFound desc = could not find container \"271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0\": container with ID starting with 271c8517c77ef6efa509b41ca6720fb6f618c081d683ea6ab1552c4a0bb282a0 not found: ID does not exist"
Jul 28 10:58:53.390782 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.390769    2560 scope.go:117] "RemoveContainer" containerID="1000dc24b2c9a4042aa42693800676144c17a6502ecc66ff1cfe48a343f22e1b"
Jul 28 10:58:53.394350 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.394329    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"]
Jul 28 10:58:53.397699 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.397682    2560 scope.go:117] "RemoveContainer" containerID="d3097639720bde7fd6bb1361da2c993367601a83712cffe049ee81a7179dd490"
Jul 28 10:58:53.401335 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.401311    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a647d-predictor-5474579676-s27ps"]
Jul 28 10:58:53.412771 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.412750    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"]
Jul 28 10:58:53.418606 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:53.418586    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a647d-predictor-784bd7c46f-s7brw"]
Jul 28 10:58:54.977287 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:54.977252    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" path="/var/lib/kubelet/pods/1194a9bf-bc05-43b9-9b6b-354a76f2eab9/volumes"
Jul 28 10:58:54.977673 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:58:54.977638    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" path="/var/lib/kubelet/pods/475ccac3-d5d6-4509-9c8d-60f7801a8f2e/volumes"
Jul 28 10:59:18.827146 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827114    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvln5/must-gather-9fzhg"]
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827374    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827392    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827405    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827411    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827421    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827427    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827433    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827438    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827444    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827449    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827458    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827463    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827469    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827474    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827479    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827484    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827489    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827494    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kube-rbac-proxy"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827500    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827505    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827512    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container"
Jul 28 10:59:18.827506 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827517    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827523    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827528    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827567    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kserve-container"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827575    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kserve-container"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827580    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kserve-container"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827586    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1194a9bf-bc05-43b9-9b6b-354a76f2eab9" containerName="kube-rbac-proxy"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827591    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="475ccac3-d5d6-4509-9c8d-60f7801a8f2e" containerName="kube-rbac-proxy"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827596    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kube-rbac-proxy"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827601    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kube-rbac-proxy"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827607    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kserve-container"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827611    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea213dbf-7200-4c6d-8f12-78e23917b867" containerName="kube-rbac-proxy"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827616    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c320d1a-ea74-423e-b59b-21aaff97675f" containerName="kserve-container"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827621    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61fa5c3-b09f-4404-bac7-3cd38f5f0924" containerName="kube-rbac-proxy"
Jul 28 10:59:18.828233 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.827627    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d66a12c5-150d-46dc-b847-11572a6b16c1" containerName="kserve-container"
Jul 28 10:59:18.831860 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.831844    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvln5/must-gather-9fzhg"
Jul 28 10:59:18.834127 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.834105    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvln5\"/\"openshift-service-ca.crt\""
Jul 28 10:59:18.834351 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.834331    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-tvln5\"/\"kube-root-ca.crt\""
Jul 28 10:59:18.834411 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.834370    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-tvln5\"/\"default-dockercfg-6kpps\""
Jul 28 10:59:18.838825 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.838789    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvln5/must-gather-9fzhg"]
Jul 28 10:59:18.999514 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.999490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trxwc\" (UniqueName: \"kubernetes.io/projected/9178255d-4008-4711-9204-7e532b19418c-kube-api-access-trxwc\") pod \"must-gather-9fzhg\" (UID: \"9178255d-4008-4711-9204-7e532b19418c\") " pod="openshift-must-gather-tvln5/must-gather-9fzhg"
Jul 28 10:59:18.999639 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:18.999522    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9178255d-4008-4711-9204-7e532b19418c-must-gather-output\") pod \"must-gather-9fzhg\" (UID: \"9178255d-4008-4711-9204-7e532b19418c\") " pod="openshift-must-gather-tvln5/must-gather-9fzhg"
Jul 28 10:59:19.100594 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:19.100532    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trxwc\" (UniqueName: \"kubernetes.io/projected/9178255d-4008-4711-9204-7e532b19418c-kube-api-access-trxwc\") pod \"must-gather-9fzhg\" (UID: \"9178255d-4008-4711-9204-7e532b19418c\") " pod="openshift-must-gather-tvln5/must-gather-9fzhg"
Jul 28 10:59:19.100594 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:19.100562    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9178255d-4008-4711-9204-7e532b19418c-must-gather-output\") pod \"must-gather-9fzhg\" (UID: \"9178255d-4008-4711-9204-7e532b19418c\") " pod="openshift-must-gather-tvln5/must-gather-9fzhg"
Jul 28 10:59:19.100941 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:19.100923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/9178255d-4008-4711-9204-7e532b19418c-must-gather-output\") pod \"must-gather-9fzhg\" (UID: \"9178255d-4008-4711-9204-7e532b19418c\") " pod="openshift-must-gather-tvln5/must-gather-9fzhg"
Jul 28 10:59:19.109567 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:19.109541    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trxwc\" (UniqueName: \"kubernetes.io/projected/9178255d-4008-4711-9204-7e532b19418c-kube-api-access-trxwc\") pod \"must-gather-9fzhg\" (UID: \"9178255d-4008-4711-9204-7e532b19418c\") " pod="openshift-must-gather-tvln5/must-gather-9fzhg"
Jul 28 10:59:19.141161 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:19.141139    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvln5/must-gather-9fzhg"
Jul 28 10:59:19.266810 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:19.266773    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvln5/must-gather-9fzhg"]
Jul 28 10:59:19.269885 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:59:19.269851    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9178255d_4008_4711_9204_7e532b19418c.slice/crio-83fa936247c085133ab27083543df13c172d225839fb25316dc26112baf5f312 WatchSource:0}: Error finding container 83fa936247c085133ab27083543df13c172d225839fb25316dc26112baf5f312: Status 404 returned error can't find the container with id 83fa936247c085133ab27083543df13c172d225839fb25316dc26112baf5f312
Jul 28 10:59:19.271736 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:19.271707    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 28 10:59:19.472862 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:19.472831    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvln5/must-gather-9fzhg" event={"ID":"9178255d-4008-4711-9204-7e532b19418c","Type":"ContainerStarted","Data":"83fa936247c085133ab27083543df13c172d225839fb25316dc26112baf5f312"}
Jul 28 10:59:20.478729 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:20.478690    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvln5/must-gather-9fzhg" event={"ID":"9178255d-4008-4711-9204-7e532b19418c","Type":"ContainerStarted","Data":"50ebca191dae949615c273cc653c4f377a8f0cdc11e6a77436e763af96b40e16"}
Jul 28 10:59:20.479202 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:20.478743    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvln5/must-gather-9fzhg" event={"ID":"9178255d-4008-4711-9204-7e532b19418c","Type":"ContainerStarted","Data":"9e0f9a06a74d2a1f405486e8513f0625ab51766b553d3798c21aa20f0add48b0"}
Jul 28 10:59:20.496911 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:20.496868    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tvln5/must-gather-9fzhg" podStartSLOduration=1.7162691890000001 podStartE2EDuration="2.496850942s" totalImagesPullingTime="780.581753ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-28 10:59:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:59:20.496286782 +0000 UTC m=+3068.071479710" watchObservedRunningTime="2026-07-28 10:59:20.496850942 +0000 UTC m=+3068.072043932"
Jul 28 10:59:21.636274 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:21.636248    2560 ???:1] "http: TLS handshake error from 10.0.140.175:51812: EOF"
Jul 28 10:59:21.642511 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:21.642482    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-t6srt_74e6f51a-e169-4372-9ba9-48189d368e93/global-pull-secret-syncer/0.log"
Jul 28 10:59:21.799976 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:21.799945    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-szf6k_0e7d9a9f-b5fe-4a2e-9737-8b9059d81eff/konnectivity-agent/0.log"
Jul 28 10:59:21.882643 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:21.882614    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-131-51.ec2.internal_2cf42ad45c184732bda36fba61d2a3c5/haproxy/0.log"
Jul 28 10:59:26.141908 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:26.141884    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2tq4s_1586e95a-6b9f-4504-8094-9a6f47b0a5fc/node-exporter/0.log"
Jul 28 10:59:26.168941 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:26.168914    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2tq4s_1586e95a-6b9f-4504-8094-9a6f47b0a5fc/kube-rbac-proxy/0.log"
Jul 28 10:59:26.195621 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:26.195595    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-2tq4s_1586e95a-6b9f-4504-8094-9a6f47b0a5fc/init-textfile/0.log"
Jul 28 10:59:28.637436 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:28.637404    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/1.log"
Jul 28 10:59:28.644411 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:28.644389    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-584dcb6b54-5568q_ecc8553b-b5a2-482e-bb37-6281729826a2/console-operator/2.log"
Jul 28 10:59:29.380350 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.380316    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"]
Jul 28 10:59:29.385824 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.385800    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.390982 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.390945    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"]
Jul 28 10:59:29.493759 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.493723    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r4h6c\" (UniqueName: \"kubernetes.io/projected/68bae8e2-a2cc-4956-a88b-e3a5896494c5-kube-api-access-r4h6c\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.494052 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.494032    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-sys\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.494214 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.494195    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-proc\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.494356 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.494340    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-podres\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.494508 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.494490    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-lib-modules\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.595816 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.595782    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-podres\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.596005 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.595831    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-lib-modules\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.596005 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.595869    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r4h6c\" (UniqueName: \"kubernetes.io/projected/68bae8e2-a2cc-4956-a88b-e3a5896494c5-kube-api-access-r4h6c\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.596005 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.595903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-sys\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.596005 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.595929    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-proc\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.596005 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.595983    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-podres\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.596005 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.595998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-lib-modules\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.596340 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.595998    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-proc\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.596340 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.596030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/68bae8e2-a2cc-4956-a88b-e3a5896494c5-sys\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.604871 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.604849    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r4h6c\" (UniqueName: \"kubernetes.io/projected/68bae8e2-a2cc-4956-a88b-e3a5896494c5-kube-api-access-r4h6c\") pod \"perf-node-gather-daemonset-vdb69\" (UID: \"68bae8e2-a2cc-4956-a88b-e3a5896494c5\") " pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.633291 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.633234    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-549c6d54cf-sg5vz_ae99e605-6566-498c-87dc-a3c0c2bf3fbe/volume-data-source-validator/0.log"
Jul 28 10:59:29.699327 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.699295    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:29.862183 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:29.861059    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"]
Jul 28 10:59:29.865807 ip-10-0-131-51 kubenswrapper[2560]: W0728 10:59:29.865776    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod68bae8e2_a2cc_4956_a88b_e3a5896494c5.slice/crio-57b2d53993b7b1e9aab2573e2b7a79bad8a93f2151cdff42f032ab5871048fc9 WatchSource:0}: Error finding container 57b2d53993b7b1e9aab2573e2b7a79bad8a93f2151cdff42f032ab5871048fc9: Status 404 returned error can't find the container with id 57b2d53993b7b1e9aab2573e2b7a79bad8a93f2151cdff42f032ab5871048fc9
Jul 28 10:59:30.519988 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:30.519960    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4gf6b_1416cd8c-7960-4cc9-8444-bc3ad41e74d2/dns/0.log"
Jul 28 10:59:30.529014 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:30.528976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69" event={"ID":"68bae8e2-a2cc-4956-a88b-e3a5896494c5","Type":"ContainerStarted","Data":"d9c76553f0cd918da7b066163797f7959145d22c04774c2196302a13d86c0feb"}
Jul 28 10:59:30.529180 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:30.529122    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69" event={"ID":"68bae8e2-a2cc-4956-a88b-e3a5896494c5","Type":"ContainerStarted","Data":"57b2d53993b7b1e9aab2573e2b7a79bad8a93f2151cdff42f032ab5871048fc9"}
Jul 28 10:59:30.529180 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:30.529167    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:30.546657 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:30.546610    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69" podStartSLOduration=1.54659514 podStartE2EDuration="1.54659514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-28 10:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-28 10:59:30.546059361 +0000 UTC m=+3078.121252282" watchObservedRunningTime="2026-07-28 10:59:30.54659514 +0000 UTC m=+3078.121788069"
Jul 28 10:59:30.548438 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:30.548414    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-4gf6b_1416cd8c-7960-4cc9-8444-bc3ad41e74d2/kube-rbac-proxy/0.log"
Jul 28 10:59:30.737113 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:30.737084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-n72pd_e3bf6896-b4a0-4bf8-b1a3-db1721465614/dns-node-resolver/0.log"
Jul 28 10:59:31.217919 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:31.216658    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5bfcf7795c-h4m2z_8bd08be2-abd3-41e9-bbee-d9ab879633ce/registry/0.log"
Jul 28 10:59:31.243202 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:31.243176    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-6zfzk_de754c06-e8e4-4e27-ad49-c4d6f1513d34/node-ca/0.log"
Jul 28 10:59:32.509939 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:32.509874    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-vswsn_fda1e514-a122-4d60-8ba6-564837522a48/serve-healthcheck-canary/0.log"
Jul 28 10:59:32.937596 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:32.937571    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk59v_39fdcf2e-356b-49df-8e08-10a12febca9e/kube-rbac-proxy/0.log"
Jul 28 10:59:32.963995 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:32.963966    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk59v_39fdcf2e-356b-49df-8e08-10a12febca9e/exporter/0.log"
Jul 28 10:59:32.991444 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:32.991422    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-gk59v_39fdcf2e-356b-49df-8e08-10a12febca9e/extractor/0.log"
Jul 28 10:59:35.687538 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:35.687506    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-6968749498-vpx4g_f88840aa-df6f-45db-b54f-253a375fa2fe/manager/0.log"
Jul 28 10:59:35.869758 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:35.869722    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_odh-model-controller-75d9cdbddf-xvh7j_2506c208-d1a3-4c69-a08b-f82ed2bb6d79/manager/0.log"
Jul 28 10:59:35.961578 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:35.961484    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-9qjn5_0ddfab1b-2429-44c8-b92f-de0de7169e82/seaweedfs/0.log"
Jul 28 10:59:36.547970 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:36.547941    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-tvln5/perf-node-gather-daemonset-vdb69"
Jul 28 10:59:41.167768 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:41.167731    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr_fab0b7f4-41fe-4684-b4fe-5497dfe59961/kube-storage-version-migrator-operator/1.log"
Jul 28 10:59:41.168555 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:41.168535    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-6f4d6fb6bc-44kqr_fab0b7f4-41fe-4684-b4fe-5497dfe59961/kube-storage-version-migrator-operator/0.log"
Jul 28 10:59:42.151823 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:42.151776    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-6h4hc_cd2a6033-2fee-4c06-9172-e067fe42263a/kube-multus/0.log"
Jul 28 10:59:42.182621 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:42.182594    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fljt_6699c75e-1572-493b-9107-6e46c03f0241/kube-multus-additional-cni-plugins/0.log"
Jul 28 10:59:42.207998 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:42.207961    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fljt_6699c75e-1572-493b-9107-6e46c03f0241/egress-router-binary-copy/0.log"
Jul 28 10:59:42.232692 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:42.232666    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fljt_6699c75e-1572-493b-9107-6e46c03f0241/cni-plugins/0.log"
Jul 28 10:59:42.258911 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:42.258882    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fljt_6699c75e-1572-493b-9107-6e46c03f0241/bond-cni-plugin/0.log"
Jul 28 10:59:42.285754 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:42.285709    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fljt_6699c75e-1572-493b-9107-6e46c03f0241/routeoverride-cni/0.log"
Jul 28 10:59:42.313257 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:42.313235    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fljt_6699c75e-1572-493b-9107-6e46c03f0241/whereabouts-cni-bincopy/0.log"
Jul 28 10:59:42.344314 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:42.344284    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2fljt_6699c75e-1572-493b-9107-6e46c03f0241/whereabouts-cni/0.log"
Jul 28 10:59:43.016906 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:43.016878    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vxtwq_ddff1dad-46bf-4209-97ff-8252a8347832/network-metrics-daemon/0.log"
Jul 28 10:59:43.042019 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:43.041989    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-vxtwq_ddff1dad-46bf-4209-97ff-8252a8347832/kube-rbac-proxy/0.log"
Jul 28 10:59:44.053231 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:44.053202    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cmbd_5ad2dd92-dcb7-4fe9-927d-bc213f250105/ovn-controller/0.log"
Jul 28 10:59:44.073352 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:44.073328    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cmbd_5ad2dd92-dcb7-4fe9-927d-bc213f250105/ovn-acl-logging/0.log"
Jul 28 10:59:44.098698 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:44.098675    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cmbd_5ad2dd92-dcb7-4fe9-927d-bc213f250105/kube-rbac-proxy-node/0.log"
Jul 28 10:59:44.123346 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:44.123315    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cmbd_5ad2dd92-dcb7-4fe9-927d-bc213f250105/kube-rbac-proxy-ovn-metrics/0.log"
Jul 28 10:59:44.147041 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:44.147018    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cmbd_5ad2dd92-dcb7-4fe9-927d-bc213f250105/northd/0.log"
Jul 28 10:59:44.172058 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:44.172032    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cmbd_5ad2dd92-dcb7-4fe9-927d-bc213f250105/nbdb/0.log"
Jul 28 10:59:44.197054 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:44.197031    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cmbd_5ad2dd92-dcb7-4fe9-927d-bc213f250105/sbdb/0.log"
Jul 28 10:59:44.246412 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:44.246386    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-9cmbd_5ad2dd92-dcb7-4fe9-927d-bc213f250105/ovnkube-controller/0.log"
Jul 28 10:59:45.641794 ip-10-0-131-51 kubenswrapper[2560]: I0728 10:59:45.641772    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-diagnostics_network-check-target-lm85v_a667ef37-25e2-454b-97e2-aa2b32f5b28a/network-check-target-container/0.log"