Jul 16 21:48:01.279562 ip-10-0-129-202 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 16 21:48:01.279575 ip-10-0-129-202 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 16 21:48:01.279584 ip-10-0-129-202 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 16 21:48:01.279910 ip-10-0-129-202 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 16 21:48:10.925131 ip-10-0-129-202 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 16 21:48:10.925152 ip-10-0-129-202 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 6f489923b5a743129b03426de35155cf --
Jul 16 21:50:00.091437 ip-10-0-129-202 systemd[1]: Starting Kubernetes Kubelet...
Jul 16 21:50:00.513483 ip-10-0-129-202 kubenswrapper[2585]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:00.513483 ip-10-0-129-202 kubenswrapper[2585]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 16 21:50:00.513483 ip-10-0-129-202 kubenswrapper[2585]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:00.513483 ip-10-0-129-202 kubenswrapper[2585]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 16 21:50:00.513483 ip-10-0-129-202 kubenswrapper[2585]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 16 21:50:00.513483 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.508475    2585 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513745    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513767    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513773    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513777    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513788    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513792    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513796    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513803    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513807    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513811    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513815    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513819    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513825    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513833    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513838    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513843    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513847    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513852    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.514492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513857    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513862    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513867    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513871    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513877    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513881    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513885    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513889    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513894    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513899    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513903    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513908    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513913    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513918    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513925    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513931    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513937    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513942    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513947    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.515369 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513969    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513976    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513981    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513986    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513990    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513995    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.513999    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514003    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514008    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514013    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514016    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514020    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514024    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514029    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514033    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514037    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514042    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514046    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514050    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514055    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.515912 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514060    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514064    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514068    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514073    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514077    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514085    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514091    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514112    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514118    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514122    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514126    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514130    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514134    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514142    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514147    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514152    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514157    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514161    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514165    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514169    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.516683 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514173    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514177    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514182    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514186    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514191    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514195    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514199    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514203    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514208    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514212    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514216    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514220    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514225    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514229    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514233    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514238    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514243    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514248    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514252    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514256    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.517655 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514261    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.514265    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517582    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517598    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517604    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517610    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517614    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517618    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517622    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517626    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517631    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517636    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517640    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517644    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517648    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517652    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517656    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517660    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517665    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.518557 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517669    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517673    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517677    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517681    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517687    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517691    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517696    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517701    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517705    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517709    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517714    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517718    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517723    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517728    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517732    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517738    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517745    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517752    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517757    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517762    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.519719 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517766    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517770    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517774    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517779    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517784    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517788    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517792    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517797    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517801    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517806    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517810    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517814    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517818    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517823    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517832    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517838    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517843    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517848    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517852    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.520807 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517857    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517861    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517866    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517870    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517874    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517878    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517882    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517886    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517891    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517895    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517900    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517904    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517911    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517917    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517921    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517926    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517930    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517936    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517940    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517945    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.521428 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517949    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517953    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517958    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517962    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517966    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517970    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517977    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517983    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517988    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517992    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.517996    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518000    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518004    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518009    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518013    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518017    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518022    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518026    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518030    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518034    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.522531 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518038    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518043    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518048    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518741    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518751    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518756    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518761    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518766    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518770    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518774    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518779    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518783    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518787    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518792    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518799    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518805    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518810    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518815    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518820    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.523921 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518825    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518831    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518836    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518841    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518846    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518851    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518856    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518861    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518865    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518871    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518877    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518882    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518887    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518891    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518895    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518900    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518904    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518908    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518912    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518917    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518921    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.524781 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518925    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518929    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518934    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518938    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518943    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518947    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518952    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518956    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518960    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518965    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518979    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518984    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518989    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518993    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.518998    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519002    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519006    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519010    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519016    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519022    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.525906 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519027    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519031    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519036    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519040    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519044    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519049    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519053    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519057    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519061    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519066    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519070    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519074    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519078    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519083    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519088    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519092    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519115    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519121    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519129    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.526595 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519134    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519139    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519144    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519148    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519152    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519157    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519162    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519166    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519171    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519176    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519180    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519184    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519189    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519193    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519197    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519202    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519206    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519210    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519215    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519220    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.527502 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519224    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519230    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.519235    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519350    2585 flags.go:64] FLAG: --address="0.0.0.0"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519363    2585 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519374    2585 flags.go:64] FLAG: --anonymous-auth="true"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519381    2585 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519390    2585 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519395    2585 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519402    2585 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519410    2585 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519415    2585 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519422    2585 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519428    2585 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519434    2585 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519439    2585 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519444    2585 flags.go:64] FLAG: --cgroup-root=""
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519448    2585 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519453    2585 flags.go:64] FLAG: --client-ca-file=""
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519460    2585 flags.go:64] FLAG: --cloud-provider="external"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519465    2585 flags.go:64] FLAG: --cluster-dns="[]"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519471    2585 flags.go:64] FLAG: --cluster-domain=""
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519475    2585 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519480    2585 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 16 21:50:00.528220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519485    2585 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519492    2585 flags.go:64] FLAG: --container-log-max-files="5"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519507    2585 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519512    2585 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519518    2585 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519523    2585 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519529    2585 flags.go:64] FLAG: --contention-profiling="false"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519534    2585 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519540    2585 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519545    2585 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519550    2585 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519555    2585 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519559    2585 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519563    2585 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519567    2585 flags.go:64] FLAG: --enable-load-reader="false"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519571    2585 flags.go:64] FLAG: --enable-server="true"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519574    2585 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519581    2585 flags.go:64] FLAG: --event-burst="100"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519587    2585 flags.go:64] FLAG: --event-qps="50"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519591    2585 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519595    2585 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519599    2585 flags.go:64] FLAG: --eviction-hard=""
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519604    2585 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519608    2585 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519612    2585 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 16 21:50:00.529094 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519615    2585 flags.go:64] FLAG: --eviction-soft=""
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519619    2585 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519623    2585 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519627    2585 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519631    2585 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519635    2585 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519638    2585 flags.go:64] FLAG: --fail-swap-on="true"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519642    2585 flags.go:64] FLAG: --feature-gates=""
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519647    2585 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519651    2585 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519655    2585 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519659    2585 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519663    2585 flags.go:64] FLAG: --healthz-port="10248"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519668    2585 flags.go:64] FLAG: --help="false"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519672    2585 flags.go:64] FLAG: --hostname-override="ip-10-0-129-202.ec2.internal"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519678    2585 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519683    2585 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519688    2585 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519693    2585 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519699    2585 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519704    2585 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519709    2585 flags.go:64] FLAG: --image-service-endpoint=""
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519714    2585 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 16 21:50:00.530945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519718    2585 flags.go:64] FLAG: --kube-api-burst="100"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519723    2585 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519730    2585 flags.go:64] FLAG: --kube-api-qps="50"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519735    2585 flags.go:64] FLAG: --kube-reserved=""
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519740    2585 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519744    2585 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519750    2585 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519754    2585 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519759    2585 flags.go:64] FLAG: --lock-file=""
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519764    2585 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519769    2585 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519774    2585 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519783    2585 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519788    2585 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519793    2585 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519798    2585 flags.go:64] FLAG: --logging-format="text"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519803    2585 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519809    2585 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519813    2585 flags.go:64] FLAG: --manifest-url=""
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519818    2585 flags.go:64] FLAG: --manifest-url-header=""
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519828    2585 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519833    2585 flags.go:64] FLAG: --max-open-files="1000000"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519840    2585 flags.go:64] FLAG: --max-pods="110"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519845    2585 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519850    2585 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 16 21:50:00.531746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519854    2585 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519861    2585 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519865    2585 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519870    2585 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519876    2585 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519890    2585 flags.go:64] FLAG: --node-status-max-images="50"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519895    2585 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519900    2585 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519905    2585 flags.go:64] FLAG: --pod-cidr=""
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519911    2585 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:4017e99d76d7b91598d3a3430cd0eba6eda8b33653addd3838fdfe3cca69c5a0"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519921    2585 flags.go:64] FLAG: --pod-manifest-path=""
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519926    2585 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519931    2585 flags.go:64] FLAG: --pods-per-core="0"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519936    2585 flags.go:64] FLAG: --port="10250"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519941    2585 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519945    2585 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-077148f6d200afd29"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519950    2585 flags.go:64] FLAG: --qos-reserved=""
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519955    2585 flags.go:64] FLAG: --read-only-port="10255"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519960    2585 flags.go:64] FLAG: --register-node="true"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519965    2585 flags.go:64] FLAG: --register-with-taints=""
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519971    2585 flags.go:64] FLAG: --registry-burst="10"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519977    2585 flags.go:64] FLAG: --registry-qps="5"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519982    2585 flags.go:64] FLAG: --reserved-cpus=""
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519986    2585 flags.go:64] FLAG: --reserved-memory=""
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519993    2585 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 16 21:50:00.604107 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.519998    2585 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 16 21:50:00.595954 ip-10-0-129-202 systemd[1]: Started Kubernetes Kubelet.
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520002    2585 flags.go:64] FLAG: --rotate-certificates="false"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520009    2585 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520015    2585 flags.go:64] FLAG: --runonce="false"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520019    2585 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520025    2585 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520030    2585 flags.go:64] FLAG: --seccomp-default="false"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520035    2585 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520040    2585 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520044    2585 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520050    2585 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520055    2585 flags.go:64] FLAG: --storage-driver-password="root"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520060    2585 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520065    2585 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520070    2585 flags.go:64] FLAG: --storage-driver-user="root"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520075    2585 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520080    2585 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520089    2585 flags.go:64] FLAG: --system-cgroups=""
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520113    2585 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520123    2585 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520128    2585 flags.go:64] FLAG: --tls-cert-file=""
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520133    2585 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520140    2585 flags.go:64] FLAG: --tls-min-version=""
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520144    2585 flags.go:64] FLAG: --tls-private-key-file=""
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520149    2585 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 16 21:50:00.605254 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520153    2585 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520159    2585 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520164    2585 flags.go:64] FLAG: --v="2"
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520171    2585 flags.go:64] FLAG: --version="false"
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520185    2585 flags.go:64] FLAG: --vmodule=""
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520191    2585 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.520196    2585 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520396    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520402    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520413    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520418    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520423    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520428    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520434    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520438    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520443    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520447    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520451    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520456    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520460    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520464    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520469    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.606181 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520473    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520477    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520484    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520488    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520492    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520496    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520501    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520506    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520510    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520515    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520519    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520523    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520527    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520531    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520536    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520541    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520545    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520549    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520553    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520560    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.607092 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520565    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520569    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520573    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520577    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520582    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520585    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520590    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520593    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520598    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520602    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520605    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520609    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520612    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520615    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520620    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520623    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520627    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520631    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520634    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520637    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.607784 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520640    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520643    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520646    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520650    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520653    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520656    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520659    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520663    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520669    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520673    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520677    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520683    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520687    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520690    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520694    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520698    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520701    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520705    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520708    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.608681 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520711    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520714    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520718    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520721    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520724    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520727    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520730    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520735    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520738    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520743    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520747    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520752    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520756    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520760    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520765    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520769    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520775    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520782    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520786    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.609372 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520791    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520795    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520799    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520804    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520809    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.520815    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.521511    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.530408    2585 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.530428    2585 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.530447    2585 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.530460    2585 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530547    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530554    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530559    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.610132 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530563    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530567    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530572    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530576    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530580    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530584    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530588    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530593    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530597    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530602    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530606    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530610    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530614    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530617    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530621    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530625    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530628    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530632    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530636    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530639    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530643    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.610925 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530653    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530657    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530660    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530664    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530668    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530671    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530675    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530678    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530682    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530686    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530690    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530695    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530700    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530704    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530709    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530712    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530715    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530718    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530722    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.612030 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530727    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530732    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530735    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530739    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530743    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530746    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530749    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530752    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530755    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530759    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530763    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530767    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530771    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530774    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530778    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530782    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530785    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530788    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530791    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.612918 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530794    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530797    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530800    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530803    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530806    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530809    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530811    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530814    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530818    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530821    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530823    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530826    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530829    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530832    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530836    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530840    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530844    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530847    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530850    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530853    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.613702 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530855    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530858    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530861    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530864    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530867    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530869    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530872    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530875    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530879    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530882    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530885    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530887    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530891    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530893    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530897    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530900    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.614554 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.530903    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.530910    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531073    2585 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531080    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531083    2585 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531087    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531090    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531093    2585 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531096    2585 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531099    2585 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531101    2585 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531105    2585 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531107    2585 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531110    2585 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 16 21:50:00.615721 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531113    2585 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531118    2585 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531122    2585 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531125    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531128    2585 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531131    2585 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531134    2585 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531137    2585 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531140    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531144    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531147    2585 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531150    2585 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531153    2585 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531156    2585 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531160    2585 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531165    2585 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531168    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531186    2585 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531189    2585 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 16 21:50:00.616478 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531192    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531195    2585 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531198    2585 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531201    2585 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531204    2585 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531207    2585 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531210    2585 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531212    2585 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531215    2585 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531218    2585 feature_gate.go:328] unrecognized feature gate: Example2
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531221    2585 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531224    2585 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531227    2585 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531230    2585 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531233    2585 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531236    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531239    2585 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531242    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531245    2585 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531248    2585 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 16 21:50:00.617381 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531252    2585 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531254    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531257    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531260    2585 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531263    2585 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531267    2585 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531270    2585 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531272    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531275    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531278    2585 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531281    2585 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531284    2585 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531286    2585 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531289    2585 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531292    2585 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531295    2585 feature_gate.go:328] unrecognized feature gate: Example
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531298    2585 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531302    2585 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531306    2585 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531311    2585 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 16 21:50:00.618110 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531314    2585 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531318    2585 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531321    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531324    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531327    2585 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531330    2585 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531333    2585 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531336    2585 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531339    2585 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531342    2585 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531345    2585 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531348    2585 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531351    2585 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531353    2585 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531356    2585 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531359    2585 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531362    2585 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531365    2585 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531374    2585 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531376    2585 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 16 21:50:00.618810 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531379    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531382    2585 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531385    2585 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531387    2585 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531391    2585 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531393    2585 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531396    2585 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:00.531400    2585 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.531405    2585 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.531551    2585 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.533908    2585 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.535225    2585 server.go:1019] "Starting client certificate rotation"
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.535340    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:50:00.619556 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.535390    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 16 21:50:00.620004 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.537627    2585 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:50:00.620004 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.539688    2585 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 16 21:50:00.620004 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.558797    2585 log.go:25] "Validated CRI v1 runtime API"
Jul 16 21:50:00.620004 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.562449    2585 log.go:25] "Validated CRI v1 image API"
Jul 16 21:50:00.620004 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.564449    2585 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 16 21:50:00.620004 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.567023    2585 fs.go:135] Filesystem UUIDs: map[3c731698-5b8b-48b4-b7c0-4fd2a6e04206:/dev/nvme0n1p4 4b3f08e8-e61a-4e25-8261-487f7ac43c60:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Jul 16 21:50:00.620004 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.567042    2585 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.572883    2585 manager.go:216] Machine: {Timestamp:2026-07-16 21:50:00.57157895 +0000 UTC m=+0.376002269 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3096092 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2de27576f827a6e35d6d16e769f247 SystemUUID:ec2de275-76f8-27a6-e35d-6d16e769f247 BootID:6f489923-b5a7-4312-9b03-426de35155cf Filesystems:[{Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:da:a4:f9:da:35 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:da:a4:f9:da:35 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:86:c6:1e:32:5d:4a Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.572996    2585 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.573098    2585 manager.go:232] Version: {KernelVersion:5.14.0-570.128.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260715-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.575933    2585 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.575969    2585 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-202.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.577274    2585 topology_manager.go:138] "Creating topology manager with none policy"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.577286    2585 container_manager_linux.go:309] "Creating device plugin manager"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.577301    2585 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.577926    2585 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.579237    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.579464    2585 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.583777    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.583920    2585 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.583950    2585 kubelet.go:486] "Attempting to sync node with API server"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.583961    2585 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.583982    2585 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 16 21:50:00.620655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.583993    2585 kubelet.go:398] "Adding apiserver pod source"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.584031    2585 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.584523    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.584540    2585 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.586885    2585 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.589374    2585 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.589406    2585 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590610    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590630    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590641    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590651    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590665    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590674    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590687    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590697    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590709    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590720    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590733    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590743    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590767    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.590776    2585 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.594812    2585 server.go:1268] "Started kubelet"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.595432    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lv64k"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.595477    2585 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.595533    2585 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.610022    2585 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.610469    2585 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.610548    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-202.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.610623    2585 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-202.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.610733    2585 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.610885    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-lv64k"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.613517    2585 server.go:310] "Adding debug handlers to kubelet server"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.617494    2585 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.618070    2585 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-202.ec2.internal.18c2e38dcefe14c5  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-202.ec2.internal,UID:ip-10-0-129-202.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-202.ec2.internal,},FirstTimestamp:2026-07-16 21:50:00.594789573 +0000 UTC m=+0.399212893,LastTimestamp:2026-07-16 21:50:00.594789573 +0000 UTC m=+0.399212893,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-202.ec2.internal,}"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.621223    2585 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.621258    2585 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 16 21:50:00.621456 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.621416    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.621878    2585 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.621905    2585 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.621919    2585 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622070    2585 reconstruct.go:97] "Volume reconstruction finished"
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622077    2585 reconciler.go:29] "Reconciler: start to sync state"
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622615    2585 factory.go:153] Registering CRI-O factory
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622635    2585 factory.go:223] Registration of the crio container factory successfully
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622704    2585 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622715    2585 factory.go:55] Registering systemd factory
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622723    2585 factory.go:223] Registration of the systemd container factory successfully
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622747    2585 factory.go:103] Registering Raw factory
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.622762    2585 manager.go:1196] Started watching for new ooms in manager
Jul 16 21:50:00.623594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.623203    2585 manager.go:349] Starting recovery of all containers
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.649156    2585 manager.go:354] Recovery completed
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.655706    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.656581    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.658738    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.658776    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.658789    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.659412    2585 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.659423    2585 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.659463    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.659445    2585 state_mem.go:36] "Initialized new in-memory state store"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.659662    2585 nodelease.go:49] "Failed to get node when trying to set owner ref to the node lease" err="nodes \"ip-10-0-129-202.ec2.internal\" not found" node="ip-10-0-129-202.ec2.internal"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.660242    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.660280    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.660291    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.661904    2585 policy_none.go:49] "None policy: Start"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.661921    2585 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.661932    2585 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 16 21:50:00.705843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.664061    2585 policy_none.go:47] "Start"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.721892    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.745004    2585 manager.go:337] "Starting Device Plugin manager"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.745059    2585 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.745076    2585 server.go:87] "Starting device plugin registration server"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.745449    2585 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.745465    2585 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.745569    2585 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.745691    2585 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.745702    2585 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.746345    2585 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.746398    2585 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.818556    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.820050    2585 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.820090    2585 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.820118    2585 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.820159    2585 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 16 21:50:00.837035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.823293    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.846307    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.847408    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.847441    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.847453    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.847480    2585 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-202.ec2.internal"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.860417    2585 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-202.ec2.internal"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.860447    2585 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-202.ec2.internal\": node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:00.914229    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.920685    2585 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"]
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.920772    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.923602    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a549896eb4d5c414c32f605c3e5efd2f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-202.ec2.internal\" (UID: \"a549896eb4d5c414c32f605c3e5efd2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.931226    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.931271    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.931286    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.934027    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.934156    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.934190    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.934941    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.934974    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.934989    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.935010    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.935036    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.935049    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.937579    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.937612    2585 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.938480    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientMemory"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.938510    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasNoDiskPressure"
Jul 16 21:50:00.943973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:00.938523    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeHasSufficientPID"
Jul 16 21:50:01.017885 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.015433    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.024918 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.024634    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a549896eb4d5c414c32f605c3e5efd2f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-202.ec2.internal\" (UID: \"a549896eb4d5c414c32f605c3e5efd2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.024918 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.024682    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/66d64bdfb7972f670b3fd86b0e11e66f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal\" (UID: \"66d64bdfb7972f670b3fd86b0e11e66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.024918 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.024715    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66d64bdfb7972f670b3fd86b0e11e66f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal\" (UID: \"66d64bdfb7972f670b3fd86b0e11e66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.024918 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.024749    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/a549896eb4d5c414c32f605c3e5efd2f-config\") pod \"kube-apiserver-proxy-ip-10-0-129-202.ec2.internal\" (UID: \"a549896eb4d5c414c32f605c3e5efd2f\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.099158 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.086899    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-202.ec2.internal\" not found" node="ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.099158 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.090126    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.099158 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.093357    2585 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-202.ec2.internal\" not found" node="ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.215930 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.116297    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.215930 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.125270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/66d64bdfb7972f670b3fd86b0e11e66f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal\" (UID: \"66d64bdfb7972f670b3fd86b0e11e66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.215930 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.125314    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66d64bdfb7972f670b3fd86b0e11e66f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal\" (UID: \"66d64bdfb7972f670b3fd86b0e11e66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.215930 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.125372    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/66d64bdfb7972f670b3fd86b0e11e66f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal\" (UID: \"66d64bdfb7972f670b3fd86b0e11e66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.215930 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.125385    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/66d64bdfb7972f670b3fd86b0e11e66f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal\" (UID: \"66d64bdfb7972f670b3fd86b0e11e66f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.217438 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.217095    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.331810 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.318146    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.425882 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.403174    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.425882 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.419132    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.579794 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.520655    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.579794 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.534822    2585 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 16 21:50:01.579794 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.534982    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:50:01.579794 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.535019    2585 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 16 21:50:01.621695 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.621657    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.625550 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.625498    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-15 21:45:00 +0000 UTC" deadline="2028-03-13 09:52:49.899912946 +0000 UTC"
Jul 16 21:50:01.625550 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.625540    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14532h2m48.274376761s"
Jul 16 21:50:01.724913 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.724809    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.825028 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:01.824988    2585 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-202.ec2.internal\" not found"
Jul 16 21:50:01.906173 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.906144    2585 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:01.922278 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.922233    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.933267 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.933241    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:50:01.935036 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.935016    2585 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal"
Jul 16 21:50:01.947162 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:01.947141    2585 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 16 21:50:02.030469 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.030443    2585 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:02.587108 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.587075    2585 apiserver.go:52] "Watching apiserver"
Jul 16 21:50:02.595648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.595266    2585 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 16 21:50:02.596744 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.596710    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-5wx6j","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l","openshift-cluster-node-tuning-operator/tuned-wn2l6","openshift-dns/node-resolver-nj98c","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal","openshift-multus/network-metrics-daemon-l9sj7","openshift-network-diagnostics/network-check-target-dhp2x","openshift-ovn-kubernetes/ovnkube-node-hnjgb","kube-system/konnectivity-agent-cms5w","kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal","openshift-image-registry/node-ca-jf8qh","openshift-multus/multus-86mjg","openshift-multus/multus-additional-cni-plugins-xwr8m","openshift-network-operator/iptables-alerter-nx42c"]
Jul 16 21:50:02.599771 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.599732    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:02.599908 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.599822    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:02.602712 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.602687    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.605631 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.605605    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.605761 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.605719    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-w5sqs\""
Jul 16 21:50:02.605859 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.605838    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.606289 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.606273    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.606400 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.606378    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 16 21:50:02.607835 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.607818    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.608287 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.608267    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.609103 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.609085    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-p24gz\""
Jul 16 21:50:02.609499 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.609484    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.610263 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.610247    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.610701 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.610685    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.611080 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.611065    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.611316 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.611301    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-vcdm4\""
Jul 16 21:50:02.612652 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.612638    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:02.612757 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.612735    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:02.614886 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.614867    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:02.616987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.616959    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 16 21:50:02.616987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.616961    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.617147 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.616973    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-7mtkd\""
Jul 16 21:50:02.617147 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.617100    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 16 21:50:02.617147 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.617107    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.617407 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.617386    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 16 21:50:02.617456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.617391    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.617456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.617439    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-l262l\""
Jul 16 21:50:02.619847 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.619827    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.622569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.622544    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.624096 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.624074    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 16 21:50:02.624979 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.624961    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.625207 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625187    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.625298 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625281    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.625357 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.625274    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:02.625419 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625399    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 16 21:50:02.625514 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625496    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 16 21:50:02.625572 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625513    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.625572 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625530    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 16 21:50:02.625676 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625515    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-6d4mc\""
Jul 16 21:50:02.625676 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625619    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.625676 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625644    2585 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 16 21:50:02.625805 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.625689    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 16 21:50:02.626205 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.626182    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 16 21:50:02.626351 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.626334    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 16 21:50:02.626463 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.626439    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2x99\""
Jul 16 21:50:02.626622 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.626604    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-958rh\""
Jul 16 21:50:02.626702 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.626627    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 16 21:50:02.627450 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.627429    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 16 21:50:02.627553 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.627429    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 16 21:50:02.628005 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.627985    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.630334 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.630274    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-wvrrk\""
Jul 16 21:50:02.630705 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.630688    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 16 21:50:02.632500 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632483    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.632551 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632509    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8ccc6eb-91cf-4474-bdbd-f8c89506deb7-konnectivity-ca\") pod \"konnectivity-agent-cms5w\" (UID: \"e8ccc6eb-91cf-4474-bdbd-f8c89506deb7\") " pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:02.632551 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632530    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cnibin\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.632645 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632550    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e9282e2-e6f4-456b-84f7-68872f2c21d0-tmp-dir\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.632645 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632579    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4xqk\" (UniqueName: \"kubernetes.io/projected/6e9282e2-e6f4-456b-84f7-68872f2c21d0-kube-api-access-f4xqk\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.632645 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632635    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:02.632743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632668    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zp6p7\" (UniqueName: \"kubernetes.io/projected/24330e43-7607-4cc4-adb6-57d6475d44a9-kube-api-access-zp6p7\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.632743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632696    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkcgz\" (UniqueName: \"kubernetes.io/projected/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-kube-api-access-dkcgz\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.632743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632711    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysctl-conf\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.632845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632742    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24330e43-7607-4cc4-adb6-57d6475d44a9-host\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.632845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632759    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.632845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632781    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-registration-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.632845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632814    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87b404d-7275-487e-aeee-fafca2282a6e-tmp\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.632845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632835    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b6fqw\" (UniqueName: \"kubernetes.io/projected/d87b404d-7275-487e-aeee-fafca2282a6e-kube-api-access-b6fqw\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632852    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4khf\" (UniqueName: \"kubernetes.io/projected/3b5ea743-a8b0-4e62-9217-8215b918bee1-kube-api-access-p4khf\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-system-cni-dir\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632898    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632941    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-socket-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.632976    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-ovn\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633005    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e9282e2-e6f4-456b-84f7-68872f2c21d0-hosts-file\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633032    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633060    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-sys-fs\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.633084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633085    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-os-release\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633109    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fq57m\" (UniqueName: \"kubernetes.io/projected/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-kube-api-access-fq57m\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633161    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-env-overrides\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633185    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-var-lib-kubelet\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633216    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-systemd\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633261    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-node-log\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633286    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovn-node-metrics-cert\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633321    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e8d7b3-b27a-4498-b51a-73813e17bed7-host-slash\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633347    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24330e43-7607-4cc4-adb6-57d6475d44a9-serviceca\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633364    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-kubelet\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633388    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-systemd-units\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633408    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-host\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.633493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633424    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-kubernetes\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.634039 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633450    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.634039 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633830    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovnkube-config\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634039 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633887    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.634039 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633940    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634039 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.633981    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovnkube-script-lib\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634039 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634010    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysconfig\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.634357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634104    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-systemd\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.634357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-log-socket\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634215    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-lib-modules\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.634357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634251    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8ccc6eb-91cf-4474-bdbd-f8c89506deb7-agent-certs\") pod \"konnectivity-agent-cms5w\" (UID: \"e8ccc6eb-91cf-4474-bdbd-f8c89506deb7\") " pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:02.634357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634294    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.634357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634325    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-slash\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634376    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-run-netns\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634431    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-var-lib-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-modprobe-d\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.634648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634525    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysctl-d\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.634648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634568    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-sys\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.634648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634602    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7252\" (UniqueName: \"kubernetes.io/projected/b5e8d7b3-b27a-4498-b51a-73813e17bed7-kube-api-access-j7252\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.634648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634633    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634659    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-device-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634693    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-etc-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634724    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-cni-netd\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634756    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d87b404d-7275-487e-aeee-fafca2282a6e-etc-tuned\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634787    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5e8d7b3-b27a-4498-b51a-73813e17bed7-iptables-alerter-script\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634814    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8xfgz\" (UniqueName: \"kubernetes.io/projected/980f3217-0513-4289-ba2b-f2c6141f803b-kube-api-access-8xfgz\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634843    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634875    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-cni-bin\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.634953 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.634903    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-run\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.643753 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.643726    2585 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 16 21:50:02.661361 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.661321    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda549896eb4d5c414c32f605c3e5efd2f.slice/crio-0a6a2d63a97ede65a453c6083199908ae034e91711eb8f3f0e75cc74355e2a84 WatchSource:0}: Error finding container 0a6a2d63a97ede65a453c6083199908ae034e91711eb8f3f0e75cc74355e2a84: Status 404 returned error can't find the container with id 0a6a2d63a97ede65a453c6083199908ae034e91711eb8f3f0e75cc74355e2a84
Jul 16 21:50:02.661632 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.661608    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66d64bdfb7972f670b3fd86b0e11e66f.slice/crio-d3275cd17d51d34edcdafc480657578e3f7255ce912a6f04532efd99bf5d3cb6 WatchSource:0}: Error finding container d3275cd17d51d34edcdafc480657578e3f7255ce912a6f04532efd99bf5d3cb6: Status 404 returned error can't find the container with id d3275cd17d51d34edcdafc480657578e3f7255ce912a6f04532efd99bf5d3cb6
Jul 16 21:50:02.666233 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.666210    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:50:02.679694 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.679656    2585 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-p7bcp"
Jul 16 21:50:02.693793 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.693769    2585 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-p7bcp"
Jul 16 21:50:02.722734 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.722697    2585 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 16 21:50:02.735264 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735225    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-system-cni-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.735369 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735272    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-k8s-cni-cncf-io\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.735369 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735295    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87b404d-7275-487e-aeee-fafca2282a6e-tmp\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.735369 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735354    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b6fqw\" (UniqueName: \"kubernetes.io/projected/d87b404d-7275-487e-aeee-fafca2282a6e-kube-api-access-b6fqw\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.735495 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735398    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4khf\" (UniqueName: \"kubernetes.io/projected/3b5ea743-a8b0-4e62-9217-8215b918bee1-kube-api-access-p4khf\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:02.735495 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735425    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-system-cni-dir\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.735495 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735450    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.735495 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735475    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-socket-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735504    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-ovn\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735547    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e9282e2-e6f4-456b-84f7-68872f2c21d0-hosts-file\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735549    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-system-cni-dir\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735583    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-ovn\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735629    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-sys-fs\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735643    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/6e9282e2-e6f4-456b-84f7-68872f2c21d0-hosts-file\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735653    2585 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735664    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-socket-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.735698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735660    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-cni-multus\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735800    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-os-release\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735811    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-sys-fs\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735842    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fq57m\" (UniqueName: \"kubernetes.io/projected/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-kube-api-access-fq57m\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735867    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-os-release\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735884    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-env-overrides\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735915    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-daemon-config\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.735977    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-var-lib-kubelet\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736009    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-systemd\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736035    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-node-log\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736065    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovn-node-metrics-cert\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736085    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-var-lib-kubelet\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736103    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cni-binary-copy\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736148    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-systemd\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736152    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-node-log\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736179    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpwk2\" (UniqueName: \"kubernetes.io/projected/b5d72dab-5005-4192-809a-00bf8632b0e6-kube-api-access-tpwk2\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736223    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e8d7b3-b27a-4498-b51a-73813e17bed7-host-slash\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736254    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24330e43-7607-4cc4-adb6-57d6475d44a9-serviceca\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736262    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b5e8d7b3-b27a-4498-b51a-73813e17bed7-host-slash\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736281    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-kubelet\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736309    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-systemd-units\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736344    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-host\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736368    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-kubelet\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736370    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-kubernetes\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-systemd-units\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736398    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-env-overrides\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736429    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736440    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-host\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736453    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-kubernetes\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736461    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovnkube-config\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736511    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-kubelet\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736597    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-hostroot\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.736981 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736639    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736688    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736715    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/24330e43-7607-4cc4-adb6-57d6475d44a9-serviceca\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736760    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736762    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovnkube-script-lib\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736816    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-tuning-conf-dir\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736864    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00df8ce7-3aa0-4336-a467-766ba77730b8-dbus\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736904    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysconfig\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736931    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-systemd\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736970    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovnkube-config\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.736987    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-systemd\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737019    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-log-socket\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737038    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysconfig\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737044    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737081    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-lib-modules\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737104    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-log-socket\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737167    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8ccc6eb-91cf-4474-bdbd-f8c89506deb7-agent-certs\") pod \"konnectivity-agent-cms5w\" (UID: \"e8ccc6eb-91cf-4474-bdbd-f8c89506deb7\") " pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:02.737824 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737194    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737238    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-slash\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737264    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-run-netns\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737286    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-var-lib-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737315    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-conf-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737345    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-modprobe-d\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737358    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-var-lib-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737392    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovnkube-script-lib\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737398    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysctl-d\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737441    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-run-netns\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737454    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-sys\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737464    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-lib-modules\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737482    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7252\" (UniqueName: \"kubernetes.io/projected/b5e8d7b3-b27a-4498-b51a-73813e17bed7-kube-api-access-j7252\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737521    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-modprobe-d\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737522    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-slash\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737565    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737555    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-sys\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.738698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737657    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysctl-d\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737686    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-device-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737716    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-device-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737728    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-etc-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737758    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-cni-netd\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737763    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-etc-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737762    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-etc-selinux\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737785    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d87b404d-7275-487e-aeee-fafca2282a6e-etc-tuned\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737812    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-cni-netd\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737831    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5e8d7b3-b27a-4498-b51a-73813e17bed7-iptables-alerter-script\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737873    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8xfgz\" (UniqueName: \"kubernetes.io/projected/980f3217-0513-4289-ba2b-f2c6141f803b-kube-api-access-8xfgz\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737904    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737930    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-cni-bin\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737958    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-cnibin\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.737985    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-os-release\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738013    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5d72dab-5005-4192-809a-00bf8632b0e6-cni-binary-copy\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.739609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738040    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-run\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738065    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00df8ce7-3aa0-4336-a467-766ba77730b8-kubelet-config\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738092    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-cni-bin\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738128    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-multus-certs\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738188    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738245    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-run-openvswitch\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738261    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-run-ovn-kubernetes\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738310    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738338    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-run\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738339    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-socket-dir-parent\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738379    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-netns\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738417    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8ccc6eb-91cf-4474-bdbd-f8c89506deb7-konnectivity-ca\") pod \"konnectivity-agent-cms5w\" (UID: \"e8ccc6eb-91cf-4474-bdbd-f8c89506deb7\") " pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738457    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cnibin\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-etc-kubernetes\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738518    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cnibin\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738526    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e9282e2-e6f4-456b-84f7-68872f2c21d0-tmp-dir\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738622    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4xqk\" (UniqueName: \"kubernetes.io/projected/6e9282e2-e6f4-456b-84f7-68872f2c21d0-kube-api-access-f4xqk\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.740356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738656    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738718    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zp6p7\" (UniqueName: \"kubernetes.io/projected/24330e43-7607-4cc4-adb6-57d6475d44a9-kube-api-access-zp6p7\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738745    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkcgz\" (UniqueName: \"kubernetes.io/projected/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-kube-api-access-dkcgz\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738773    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-cni-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738801    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysctl-conf\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738916    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24330e43-7607-4cc4-adb6-57d6475d44a9-host\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738927    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e9282e2-e6f4-456b-84f7-68872f2c21d0-tmp-dir\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738946    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.738979    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-registration-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.739042    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e8ccc6eb-91cf-4474-bdbd-f8c89506deb7-konnectivity-ca\") pod \"konnectivity-agent-cms5w\" (UID: \"e8ccc6eb-91cf-4474-bdbd-f8c89506deb7\") " pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.739085    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/980f3217-0513-4289-ba2b-f2c6141f803b-registration-dir\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.739135    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.739411    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d87b404d-7275-487e-aeee-fafca2282a6e-etc-sysctl-conf\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.739474    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-host-cni-bin\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.739497    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs podName:3b5ea743-a8b0-4e62-9217-8215b918bee1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:03.239473332 +0000 UTC m=+3.043896656 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs") pod "network-metrics-daemon-l9sj7" (UID: "3b5ea743-a8b0-4e62-9217-8215b918bee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.739537    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/24330e43-7607-4cc4-adb6-57d6475d44a9-host\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.739821    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.740989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.740106    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d87b404d-7275-487e-aeee-fafca2282a6e-tmp\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.741590 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.740226    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b5e8d7b3-b27a-4498-b51a-73813e17bed7-iptables-alerter-script\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.741590 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.740244    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d87b404d-7275-487e-aeee-fafca2282a6e-etc-tuned\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.741590 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.740346    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-ovn-node-metrics-cert\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.741590 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.740470    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e8ccc6eb-91cf-4474-bdbd-f8c89506deb7-agent-certs\") pod \"konnectivity-agent-cms5w\" (UID: \"e8ccc6eb-91cf-4474-bdbd-f8c89506deb7\") " pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:02.744854 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.744811    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:02.744854 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.744832    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:02.744854 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.744845    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rc7hz for pod openshift-network-diagnostics/network-check-target-dhp2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:02.744965 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.744893    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz podName:4a203a4e-09f9-4ede-927a-38ccaad319b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:03.244878294 +0000 UTC m=+3.049301599 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-rc7hz" (UniqueName: "kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz") pod "network-check-target-dhp2x" (UID: "4a203a4e-09f9-4ede-927a-38ccaad319b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:02.747397 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.747366    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fq57m\" (UniqueName: \"kubernetes.io/projected/f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b-kube-api-access-fq57m\") pod \"multus-additional-cni-plugins-xwr8m\" (UID: \"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b\") " pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.747867 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.747849    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7252\" (UniqueName: \"kubernetes.io/projected/b5e8d7b3-b27a-4498-b51a-73813e17bed7-kube-api-access-j7252\") pod \"iptables-alerter-nx42c\" (UID: \"b5e8d7b3-b27a-4498-b51a-73813e17bed7\") " pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.749387 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.749310    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b6fqw\" (UniqueName: \"kubernetes.io/projected/d87b404d-7275-487e-aeee-fafca2282a6e-kube-api-access-b6fqw\") pod \"tuned-wn2l6\" (UID: \"d87b404d-7275-487e-aeee-fafca2282a6e\") " pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.751825 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.751785    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkcgz\" (UniqueName: \"kubernetes.io/projected/aa83eae5-eb8c-440d-98ba-5e8f8b3714b2-kube-api-access-dkcgz\") pod \"ovnkube-node-hnjgb\" (UID: \"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2\") " pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.752138 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.752113    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4khf\" (UniqueName: \"kubernetes.io/projected/3b5ea743-a8b0-4e62-9217-8215b918bee1-kube-api-access-p4khf\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:02.752235 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.752119    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4xqk\" (UniqueName: \"kubernetes.io/projected/6e9282e2-e6f4-456b-84f7-68872f2c21d0-kube-api-access-f4xqk\") pod \"node-resolver-nj98c\" (UID: \"6e9282e2-e6f4-456b-84f7-68872f2c21d0\") " pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.753743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.753721    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8xfgz\" (UniqueName: \"kubernetes.io/projected/980f3217-0513-4289-ba2b-f2c6141f803b-kube-api-access-8xfgz\") pod \"aws-ebs-csi-driver-node-7h77l\" (UID: \"980f3217-0513-4289-ba2b-f2c6141f803b\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.755131 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.755113    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zp6p7\" (UniqueName: \"kubernetes.io/projected/24330e43-7607-4cc4-adb6-57d6475d44a9-kube-api-access-zp6p7\") pod \"node-ca-jf8qh\" (UID: \"24330e43-7607-4cc4-adb6-57d6475d44a9\") " pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.824962 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.824904    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal" event={"ID":"66d64bdfb7972f670b3fd86b0e11e66f","Type":"ContainerStarted","Data":"d3275cd17d51d34edcdafc480657578e3f7255ce912a6f04532efd99bf5d3cb6"}
Jul 16 21:50:02.825898 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.825872    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal" event={"ID":"a549896eb4d5c414c32f605c3e5efd2f","Type":"ContainerStarted","Data":"0a6a2d63a97ede65a453c6083199908ae034e91711eb8f3f0e75cc74355e2a84"}
Jul 16 21:50:02.840417 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840372    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-cni-multus\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840417 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840414    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-daemon-config\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840657 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840435    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpwk2\" (UniqueName: \"kubernetes.io/projected/b5d72dab-5005-4192-809a-00bf8632b0e6-kube-api-access-tpwk2\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840657 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840456    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-kubelet\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840657 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840502    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-hostroot\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840657 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840512    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-kubelet\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840657 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840507    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-cni-multus\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840657 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840582    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-hostroot\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840664    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00df8ce7-3aa0-4336-a467-766ba77730b8-dbus\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840704    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-conf-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840739    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-cnibin\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840757    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-os-release\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840773    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5d72dab-5005-4192-809a-00bf8632b0e6-cni-binary-copy\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840794    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00df8ce7-3aa0-4336-a467-766ba77730b8-kubelet-config\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840817    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-cni-bin\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840841    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-multus-certs\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840842    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/00df8ce7-3aa0-4336-a467-766ba77730b8-dbus\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840854    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-cnibin\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840870    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-var-lib-cni-bin\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-conf-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.840915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840892    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/00df8ce7-3aa0-4336-a467-766ba77730b8-kubelet-config\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840916    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-multus-certs\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840932    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-os-release\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840931    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840963    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-socket-dir-parent\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.840979    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-netns\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841000    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-etc-kubernetes\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.841023    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841035    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-socket-dir-parent\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841041    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-cni-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841051    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-netns\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841066    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-system-cni-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841072    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-etc-kubernetes\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:02.841090    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret podName:00df8ce7-3aa0-4336-a467-766ba77730b8 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:03.341070904 +0000 UTC m=+3.145494228 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret") pod "global-pull-secret-syncer-5wx6j" (UID: "00df8ce7-3aa0-4336-a467-766ba77730b8") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841107    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-system-cni-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841134    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-cni-dir\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841155    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-k8s-cni-cncf-io\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841154    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b5d72dab-5005-4192-809a-00bf8632b0e6-multus-daemon-config\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.841485 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841199    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b5d72dab-5005-4192-809a-00bf8632b0e6-host-run-k8s-cni-cncf-io\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.842022 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.841310    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b5d72dab-5005-4192-809a-00bf8632b0e6-cni-binary-copy\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.853545 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.853517    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpwk2\" (UniqueName: \"kubernetes.io/projected/b5d72dab-5005-4192-809a-00bf8632b0e6-kube-api-access-tpwk2\") pod \"multus-86mjg\" (UID: \"b5d72dab-5005-4192-809a-00bf8632b0e6\") " pod="openshift-multus/multus-86mjg"
Jul 16 21:50:02.901319 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.901292    2585 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:02.927984 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.927949    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-jf8qh"
Jul 16 21:50:02.939601 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.939571    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-wn2l6"
Jul 16 21:50:02.939892 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.939859    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24330e43_7607_4cc4_adb6_57d6475d44a9.slice/crio-54bf480053a5861c799ca53460669a974f1601587abf2fe0a076755e0f890f27 WatchSource:0}: Error finding container 54bf480053a5861c799ca53460669a974f1601587abf2fe0a076755e0f890f27: Status 404 returned error can't find the container with id 54bf480053a5861c799ca53460669a974f1601587abf2fe0a076755e0f890f27
Jul 16 21:50:02.948115 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.948091    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd87b404d_7275_487e_aeee_fafca2282a6e.slice/crio-1e167fea3e47cef34c0f8ae62bfac56b0db10e90b681590fcf015dab924d931f WatchSource:0}: Error finding container 1e167fea3e47cef34c0f8ae62bfac56b0db10e90b681590fcf015dab924d931f: Status 404 returned error can't find the container with id 1e167fea3e47cef34c0f8ae62bfac56b0db10e90b681590fcf015dab924d931f
Jul 16 21:50:02.960053 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.960029    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-nj98c"
Jul 16 21:50:02.968851 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.968803    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-nx42c"
Jul 16 21:50:02.969195 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.969161    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e9282e2_e6f4_456b_84f7_68872f2c21d0.slice/crio-e813b7589ce4b075d09aaad4f2ec172b9af7c1c28747a1607e93f836e0760410 WatchSource:0}: Error finding container e813b7589ce4b075d09aaad4f2ec172b9af7c1c28747a1607e93f836e0760410: Status 404 returned error can't find the container with id e813b7589ce4b075d09aaad4f2ec172b9af7c1c28747a1607e93f836e0760410
Jul 16 21:50:02.976054 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.976034    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:02.978044 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.978017    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5e8d7b3_b27a_4498_b51a_73813e17bed7.slice/crio-dc1a8407f87405937317e8e037f1860ad5bbbe40317bef602c18af90021b6ed3 WatchSource:0}: Error finding container dc1a8407f87405937317e8e037f1860ad5bbbe40317bef602c18af90021b6ed3: Status 404 returned error can't find the container with id dc1a8407f87405937317e8e037f1860ad5bbbe40317bef602c18af90021b6ed3
Jul 16 21:50:02.981505 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.981484    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-xwr8m"
Jul 16 21:50:02.987542 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.987511    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8ccc6eb_91cf_4474_bdbd_f8c89506deb7.slice/crio-f0ecfb17077caa2bbad153e9c492177e024fe0a374b24257b1d4ca3612dbb1d7 WatchSource:0}: Error finding container f0ecfb17077caa2bbad153e9c492177e024fe0a374b24257b1d4ca3612dbb1d7: Status 404 returned error can't find the container with id f0ecfb17077caa2bbad153e9c492177e024fe0a374b24257b1d4ca3612dbb1d7
Jul 16 21:50:02.987877 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.987855    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l"
Jul 16 21:50:02.991642 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.991594    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7eaefc7_e67b_4ee3_857d_dc0f6deabf2b.slice/crio-cf36a2b840630166f04a20c7e05c4cbc4f1b9c5aed020cd63eb420a5a9b40512 WatchSource:0}: Error finding container cf36a2b840630166f04a20c7e05c4cbc4f1b9c5aed020cd63eb420a5a9b40512: Status 404 returned error can't find the container with id cf36a2b840630166f04a20c7e05c4cbc4f1b9c5aed020cd63eb420a5a9b40512
Jul 16 21:50:02.994472 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.994399    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:02.998690 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:02.998537    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod980f3217_0513_4289_ba2b_f2c6141f803b.slice/crio-6d92b7bccb8ff80a6a771ea4f22bbf3562522211aae8925214155490bf264dbf WatchSource:0}: Error finding container 6d92b7bccb8ff80a6a771ea4f22bbf3562522211aae8925214155490bf264dbf: Status 404 returned error can't find the container with id 6d92b7bccb8ff80a6a771ea4f22bbf3562522211aae8925214155490bf264dbf
Jul 16 21:50:02.999340 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:02.999320    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-86mjg"
Jul 16 21:50:03.005028 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:03.005004    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa83eae5_eb8c_440d_98ba_5e8f8b3714b2.slice/crio-6efa5127a4b527a87139d15fc375097264f05f9230a8e3eec71206cc97725129 WatchSource:0}: Error finding container 6efa5127a4b527a87139d15fc375097264f05f9230a8e3eec71206cc97725129: Status 404 returned error can't find the container with id 6efa5127a4b527a87139d15fc375097264f05f9230a8e3eec71206cc97725129
Jul 16 21:50:03.009787 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:03.009759    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5d72dab_5005_4192_809a_00bf8632b0e6.slice/crio-41a1d17b9b4b55c70856e3b17d9f4e7cfa62b4ee61073d72b343e044bc8714fe WatchSource:0}: Error finding container 41a1d17b9b4b55c70856e3b17d9f4e7cfa62b4ee61073d72b343e044bc8714fe: Status 404 returned error can't find the container with id 41a1d17b9b4b55c70856e3b17d9f4e7cfa62b4ee61073d72b343e044bc8714fe
Jul 16 21:50:03.075344 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.075295    2585 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 16 21:50:03.243630 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.243576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:03.243821 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.243780    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:03.243897 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.243852    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs podName:3b5ea743-a8b0-4e62-9217-8215b918bee1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:04.243836215 +0000 UTC m=+4.048259521 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs") pod "network-metrics-daemon-l9sj7" (UID: "3b5ea743-a8b0-4e62-9217-8215b918bee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:03.344689 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.344583    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:03.344689 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.344673    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:03.345036 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.344848    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:03.345036 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.344872    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:03.345036 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.344887    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rc7hz for pod openshift-network-diagnostics/network-check-target-dhp2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:03.345036 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.344970    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz podName:4a203a4e-09f9-4ede-927a-38ccaad319b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:04.344934591 +0000 UTC m=+4.149357921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-rc7hz" (UniqueName: "kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz") pod "network-check-target-dhp2x" (UID: "4a203a4e-09f9-4ede-927a-38ccaad319b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:03.345311 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.345056    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:03.345311 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.345103    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret podName:00df8ce7-3aa0-4336-a467-766ba77730b8 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:04.345082522 +0000 UTC m=+4.149505830 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret") pod "global-pull-secret-syncer-5wx6j" (UID: "00df8ce7-3aa0-4336-a467-766ba77730b8") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:03.698808 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.698587    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:45:02 +0000 UTC" deadline="2028-03-01 05:41:53.528944352 +0000 UTC"
Jul 16 21:50:03.698808 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.698684    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14239h51m49.830264794s"
Jul 16 21:50:03.821906 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.821080    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:03.821906 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.821253    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:03.821906 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.821733    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:03.821906 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:03.821851    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:03.837455 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.837409    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l" event={"ID":"980f3217-0513-4289-ba2b-f2c6141f803b","Type":"ContainerStarted","Data":"6d92b7bccb8ff80a6a771ea4f22bbf3562522211aae8925214155490bf264dbf"}
Jul 16 21:50:03.864322 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.864267    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" event={"ID":"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b","Type":"ContainerStarted","Data":"cf36a2b840630166f04a20c7e05c4cbc4f1b9c5aed020cd63eb420a5a9b40512"}
Jul 16 21:50:03.871190 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.871114    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cms5w" event={"ID":"e8ccc6eb-91cf-4474-bdbd-f8c89506deb7","Type":"ContainerStarted","Data":"f0ecfb17077caa2bbad153e9c492177e024fe0a374b24257b1d4ca3612dbb1d7"}
Jul 16 21:50:03.880250 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.880203    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nj98c" event={"ID":"6e9282e2-e6f4-456b-84f7-68872f2c21d0","Type":"ContainerStarted","Data":"e813b7589ce4b075d09aaad4f2ec172b9af7c1c28747a1607e93f836e0760410"}
Jul 16 21:50:03.883647 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.883608    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wn2l6" event={"ID":"d87b404d-7275-487e-aeee-fafca2282a6e","Type":"ContainerStarted","Data":"1e167fea3e47cef34c0f8ae62bfac56b0db10e90b681590fcf015dab924d931f"}
Jul 16 21:50:03.887433 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.887334    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx42c" event={"ID":"b5e8d7b3-b27a-4498-b51a-73813e17bed7","Type":"ContainerStarted","Data":"dc1a8407f87405937317e8e037f1860ad5bbbe40317bef602c18af90021b6ed3"}
Jul 16 21:50:03.898514 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.898417    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jf8qh" event={"ID":"24330e43-7607-4cc4-adb6-57d6475d44a9","Type":"ContainerStarted","Data":"54bf480053a5861c799ca53460669a974f1601587abf2fe0a076755e0f890f27"}
Jul 16 21:50:03.902186 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.902079    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86mjg" event={"ID":"b5d72dab-5005-4192-809a-00bf8632b0e6","Type":"ContainerStarted","Data":"41a1d17b9b4b55c70856e3b17d9f4e7cfa62b4ee61073d72b343e044bc8714fe"}
Jul 16 21:50:03.919820 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:03.919773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"6efa5127a4b527a87139d15fc375097264f05f9230a8e3eec71206cc97725129"}
Jul 16 21:50:04.252314 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:04.252263    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:04.252529 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.252490    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:04.252593 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.252562    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs podName:3b5ea743-a8b0-4e62-9217-8215b918bee1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:06.252540386 +0000 UTC m=+6.056963710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs") pod "network-metrics-daemon-l9sj7" (UID: "3b5ea743-a8b0-4e62-9217-8215b918bee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:04.353077 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:04.353030    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:04.353287 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:04.353144    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:04.353380 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.353334    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:04.353380 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.353361    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:04.353380 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.353376    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rc7hz for pod openshift-network-diagnostics/network-check-target-dhp2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:04.353539 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.353447    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz podName:4a203a4e-09f9-4ede-927a-38ccaad319b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:06.353424747 +0000 UTC m=+6.157848053 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-rc7hz" (UniqueName: "kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz") pod "network-check-target-dhp2x" (UID: "4a203a4e-09f9-4ede-927a-38ccaad319b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:04.353945 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.353917    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:04.354054 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.354027    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret podName:00df8ce7-3aa0-4336-a467-766ba77730b8 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:06.354009238 +0000 UTC m=+6.158432548 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret") pod "global-pull-secret-syncer-5wx6j" (UID: "00df8ce7-3aa0-4336-a467-766ba77730b8") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:04.699857 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:04.699751    2585 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-15 21:45:02 +0000 UTC" deadline="2028-04-18 04:36:01.7459478 +0000 UTC"
Jul 16 21:50:04.699857 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:04.699801    2585 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15390h45m57.046152377s"
Jul 16 21:50:04.823250 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:04.823211    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:04.823466 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:04.823377    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:05.821271 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:05.820453    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:05.821271 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:05.820592    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:05.821271 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:05.820974    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:05.821271 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:05.821090    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:06.268987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:06.268352    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:06.268987 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.268539    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:06.268987 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.268620    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs podName:3b5ea743-a8b0-4e62-9217-8215b918bee1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:10.268598389 +0000 UTC m=+10.073021708 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs") pod "network-metrics-daemon-l9sj7" (UID: "3b5ea743-a8b0-4e62-9217-8215b918bee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:06.369452 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:06.369411    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:06.369636 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:06.369501    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:06.369714 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.369648    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:06.369714 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.369713    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret podName:00df8ce7-3aa0-4336-a467-766ba77730b8 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:10.369693485 +0000 UTC m=+10.174116793 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret") pod "global-pull-secret-syncer-5wx6j" (UID: "00df8ce7-3aa0-4336-a467-766ba77730b8") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:06.370164 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.370143    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:06.370259 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.370172    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:06.370259 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.370185    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rc7hz for pod openshift-network-diagnostics/network-check-target-dhp2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:06.370259 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.370237    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz podName:4a203a4e-09f9-4ede-927a-38ccaad319b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:10.370218358 +0000 UTC m=+10.174641681 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-rc7hz" (UniqueName: "kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz") pod "network-check-target-dhp2x" (UID: "4a203a4e-09f9-4ede-927a-38ccaad319b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:06.823417 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:06.823383    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:06.823897 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:06.823527    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:07.820898 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:07.820833    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:07.821131 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:07.820977    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:07.821647 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:07.821372    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:07.821647 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:07.821479    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:08.821347 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:08.821156    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:08.821347 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:08.821309    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:09.821164 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:09.820425    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:09.821164 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:09.820588    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:09.821164 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:09.820984    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:09.821164 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:09.821121    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:10.306070 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:10.306016    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:10.306272 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.306246    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:10.306341 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.306327    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs podName:3b5ea743-a8b0-4e62-9217-8215b918bee1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:18.306305941 +0000 UTC m=+18.110729257 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs") pod "network-metrics-daemon-l9sj7" (UID: "3b5ea743-a8b0-4e62-9217-8215b918bee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:10.406859 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:10.406818    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:10.407061 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:10.406917    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:10.407061 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.407013    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:10.407061 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.407030    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:10.407061 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.407041    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:10.407061 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.407057    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rc7hz for pod openshift-network-diagnostics/network-check-target-dhp2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:10.407265 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.407093    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret podName:00df8ce7-3aa0-4336-a467-766ba77730b8 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:18.40707375 +0000 UTC m=+18.211497076 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret") pod "global-pull-secret-syncer-5wx6j" (UID: "00df8ce7-3aa0-4336-a467-766ba77730b8") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:10.407265 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.407129    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz podName:4a203a4e-09f9-4ede-927a-38ccaad319b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:18.407118836 +0000 UTC m=+18.211542152 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-rc7hz" (UniqueName: "kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz") pod "network-check-target-dhp2x" (UID: "4a203a4e-09f9-4ede-927a-38ccaad319b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:10.822029 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:10.821991    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:10.822421 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:10.822151    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:11.820426 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:11.820383    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:11.820614 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:11.820429    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:11.820614 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:11.820537    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:11.820730 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:11.820641    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:12.820828 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:12.820410    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:12.820828 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:12.820549    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:13.820962 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:13.820926    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:13.820962 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:13.820972    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:13.821410 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:13.821086    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:13.821410 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:13.821234    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:14.820759 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:14.820717    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:14.820924 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:14.820874    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:15.821002 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:15.820956    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:15.821533 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:15.820956    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:15.821533 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:15.821094    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:15.821533 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:15.821214    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:16.821334 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:16.821294    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:16.821761 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:16.821436    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:17.820764 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:17.820694    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:17.820949 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:17.820840    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:17.820949 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:17.820894    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:17.821087 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:17.821015    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:18.368316 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:18.368274    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:18.368850 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.368464    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:18.368850 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.368554    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs podName:3b5ea743-a8b0-4e62-9217-8215b918bee1 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:34.368531289 +0000 UTC m=+34.172954611 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs") pod "network-metrics-daemon-l9sj7" (UID: "3b5ea743-a8b0-4e62-9217-8215b918bee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:18.469460 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:18.469427    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:18.469631 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:18.469488    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:18.469631 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.469611    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:18.469723 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.469692    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret podName:00df8ce7-3aa0-4336-a467-766ba77730b8 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:34.469670261 +0000 UTC m=+34.274093585 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret") pod "global-pull-secret-syncer-5wx6j" (UID: "00df8ce7-3aa0-4336-a467-766ba77730b8") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:18.469723 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.469620    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:18.469723 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.469723    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:18.469845 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.469733    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rc7hz for pod openshift-network-diagnostics/network-check-target-dhp2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:18.469845 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.469776    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz podName:4a203a4e-09f9-4ede-927a-38ccaad319b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:34.469765045 +0000 UTC m=+34.274188349 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-rc7hz" (UniqueName: "kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz") pod "network-check-target-dhp2x" (UID: "4a203a4e-09f9-4ede-927a-38ccaad319b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:18.820364 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:18.820322    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:18.820532 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:18.820477    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:19.820513 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:19.820469    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:19.821012 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:19.820480    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:19.821012 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:19.820624    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:19.821012 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:19.820694    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:20.822490 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:20.822446    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:20.822956 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:20.822927    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:21.821246 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:21.821189    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:21.821451 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:21.821191    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:21.821451 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:21.821333    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:21.821451 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:21.821431    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:22.821339 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:22.821296    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:22.821826 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:22.821440    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:23.821082 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:23.820762    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:23.821282 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:23.820783    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:23.821282 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:23.821180    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:23.821396 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:23.821276    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:23.969844 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:23.968619    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal" event={"ID":"a549896eb4d5c414c32f605c3e5efd2f","Type":"ContainerStarted","Data":"8ce0e94c4b12ae8d22986b93344ce755962e4a84acd2a30be2ab17066e02946c"}
Jul 16 21:50:23.973844 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:23.972228    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-86mjg" event={"ID":"b5d72dab-5005-4192-809a-00bf8632b0e6","Type":"ContainerStarted","Data":"31d61146b0f717f24705d58b86022b137f421fb65fa2fb7a416e4f51ad81a7e3"}
Jul 16 21:50:23.978118 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:23.977974    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"18761a18edd539120e99e2511452aae8da71223a3185e02da3b431b14781cfd8"}
Jul 16 21:50:23.978118 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:23.978086    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"41d5dd7123b4483bb739c9f404382fea9cb424d5d4fd53a8c557a173fa130dd2"}
Jul 16 21:50:23.981476 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:23.980903    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-wn2l6" event={"ID":"d87b404d-7275-487e-aeee-fafca2282a6e","Type":"ContainerStarted","Data":"eaa5cda66a9a573fb371ba20ad7b1727e2b4dc077a1f0e779c380bb83c79a79f"}
Jul 16 21:50:23.987793 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:23.987725    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-202.ec2.internal" podStartSLOduration=21.987705191 podStartE2EDuration="21.987705191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:23.98683084 +0000 UTC m=+23.791254166" watchObservedRunningTime="2026-07-16 21:50:23.987705191 +0000 UTC m=+23.792128522"
Jul 16 21:50:24.009916 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.009862    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-wn2l6" podStartSLOduration=2.678073048 podStartE2EDuration="23.009845403s" totalImagesPullingTime="20.331772355s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:24.009625281 +0000 UTC m=+23.814048607" watchObservedRunningTime="2026-07-16 21:50:24.009845403 +0000 UTC m=+23.814268729"
Jul 16 21:50:24.821008 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.820967    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:24.821239 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:24.821152    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:24.986076 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.986037    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"bef4d6e89071bfa76e8c5bf11ae12244c7b4d41dc1cfe1effc6155eb891cabca"}
Jul 16 21:50:24.986889 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.986142    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"9e9ee9c426988d1f7a02ff8900596565991f896ad7b079c0f80b44c8dd23a8ba"}
Jul 16 21:50:24.986889 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.986188    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"a8fb9112860a970b16577dedd85c2f28983b715fd53bf92de443f9fd811f46a4"}
Jul 16 21:50:24.986889 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.986230    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"13f3907b9e5030f52905d9a15dc1c08249d2da0e9a8d96c3da2f5bccefcbfcd2"}
Jul 16 21:50:24.987499 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.987475    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l" event={"ID":"980f3217-0513-4289-ba2b-f2c6141f803b","Type":"ContainerStarted","Data":"fc9d3938b775d9a53c822b2314f016cdffa6f1e22a8c48c054abbed377a19b40"}
Jul 16 21:50:24.988959 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.988928    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b" containerID="5001059aec6195dc94c0a69934e1bd89bf1287f397e111d849f177ae541511f7" exitCode=0
Jul 16 21:50:24.989128 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.989011    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" event={"ID":"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b","Type":"ContainerDied","Data":"5001059aec6195dc94c0a69934e1bd89bf1287f397e111d849f177ae541511f7"}
Jul 16 21:50:24.992012 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.991981    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-cms5w" event={"ID":"e8ccc6eb-91cf-4474-bdbd-f8c89506deb7","Type":"ContainerStarted","Data":"b9168395911af0d8ef17fe831cca8c4bfbee0b03e56ded75cf89920f44528895"}
Jul 16 21:50:24.993659 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.993635    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-nj98c" event={"ID":"6e9282e2-e6f4-456b-84f7-68872f2c21d0","Type":"ContainerStarted","Data":"be58fa527c38ae557eb821d0632d950e4ce5d83282cf5d150e4ca2ccebed4a03"}
Jul 16 21:50:24.995086 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.995064    2585 generic.go:356] "Generic (PLEG): container finished" podID="66d64bdfb7972f670b3fd86b0e11e66f" containerID="0a97c4d74d0f2640b1ea7d9430c492e17ccec410a8dd9feb7258aca4cf7fdc77" exitCode=0
Jul 16 21:50:24.995190 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.995124    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal" event={"ID":"66d64bdfb7972f670b3fd86b0e11e66f","Type":"ContainerDied","Data":"0a97c4d74d0f2640b1ea7d9430c492e17ccec410a8dd9feb7258aca4cf7fdc77"}
Jul 16 21:50:24.996786 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.996764    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-nx42c" event={"ID":"b5e8d7b3-b27a-4498-b51a-73813e17bed7","Type":"ContainerStarted","Data":"ab3b33ca6a9599214708df9297f09003e2b587c69ebea4e2e6575867df3b4a9b"}
Jul 16 21:50:24.998135 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:24.998107    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-jf8qh" event={"ID":"24330e43-7607-4cc4-adb6-57d6475d44a9","Type":"ContainerStarted","Data":"8a87fbd764d323c4e65f1b852c5b4c917d13ea7bbc18d318c8e2e7e6e7ab028e"}
Jul 16 21:50:25.028337 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.028047    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-86mjg" podStartSLOduration=3.541121291 podStartE2EDuration="24.028029396s" totalImagesPullingTime="20.486908105s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:24.029555156 +0000 UTC m=+23.833978482" watchObservedRunningTime="2026-07-16 21:50:25.028029396 +0000 UTC m=+24.832452723"
Jul 16 21:50:25.045166 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.045092    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-cms5w" podStartSLOduration=3.902841866 podStartE2EDuration="24.045069253s" totalImagesPullingTime="20.142227387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:25.044418322 +0000 UTC m=+24.848841651" watchObservedRunningTime="2026-07-16 21:50:25.045069253 +0000 UTC m=+24.849492579"
Jul 16 21:50:25.079255 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.079205    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-jf8qh" podStartSLOduration=3.740199843 podStartE2EDuration="24.079186421s" totalImagesPullingTime="20.338986578s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:25.062370104 +0000 UTC m=+24.866793431" watchObservedRunningTime="2026-07-16 21:50:25.079186421 +0000 UTC m=+24.883609747"
Jul 16 21:50:25.079424 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.079406    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-nx42c" podStartSLOduration=3.778065123 podStartE2EDuration="24.079399104s" totalImagesPullingTime="20.301333981s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:25.079055079 +0000 UTC m=+24.883478404" watchObservedRunningTime="2026-07-16 21:50:25.079399104 +0000 UTC m=+24.883822430"
Jul 16 21:50:25.137442 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.137388    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-nj98c" podStartSLOduration=3.853443329 podStartE2EDuration="24.137371129s" totalImagesPullingTime="20.2839278s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:25.137115511 +0000 UTC m=+24.941538829" watchObservedRunningTime="2026-07-16 21:50:25.137371129 +0000 UTC m=+24.941794454"
Jul 16 21:50:25.260152 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.260109    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:25.276226 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.276173    2585 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 16 21:50:25.336888 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.336788    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:25.758074 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.757940    2585 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-16T21:50:25.276206865Z","UUID":"e498b6eb-e547-43ad-865a-549f6f30eacf","Handler":null,"Name":"","Endpoint":""}
Jul 16 21:50:25.760662 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.760209    2585 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 16 21:50:25.760662 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.760282    2585 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 16 21:50:25.820830 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.820795    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:25.821059 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:25.820916    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:25.821059 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:25.820973    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:25.821258 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:25.821138    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:26.003314 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:26.003276    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l" event={"ID":"980f3217-0513-4289-ba2b-f2c6141f803b","Type":"ContainerStarted","Data":"abc3cdc9fcaa295fdcd71ac821e2cda459f0c36730ac9ffefb32117880f5332a"}
Jul 16 21:50:26.005804 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:26.005736    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal" event={"ID":"66d64bdfb7972f670b3fd86b0e11e66f","Type":"ContainerStarted","Data":"ba63914ca1aab5e686aa610f9b81f4bfe6e3294c3c00e1d84a9237c433edcb84"}
Jul 16 21:50:26.039791 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:26.039684    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-202.ec2.internal" podStartSLOduration=24.03966949 podStartE2EDuration="24.03966949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:26.039488091 +0000 UTC m=+25.843911417" watchObservedRunningTime="2026-07-16 21:50:26.03966949 +0000 UTC m=+25.844092815"
Jul 16 21:50:26.280488 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:26.280448    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:26.281325 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:26.281299    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:26.823955 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:26.823925    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:26.824145 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:26.824037    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:27.015687 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.015637    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"4e31dba74e08b619d962617dfd37f9f23f66a24e12b4fa896deb864474672c97"}
Jul 16 21:50:27.018288 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.018237    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l" event={"ID":"980f3217-0513-4289-ba2b-f2c6141f803b","Type":"ContainerStarted","Data":"3e0fcb0a7980c6e4cae47c9221762c72136fbd2d5c698ccbae144ddecf05addf"}
Jul 16 21:50:27.018785 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.018730    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:27.019276 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.019257    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-cms5w"
Jul 16 21:50:27.038441 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.038377    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7h77l" podStartSLOduration=3.706171627 podStartE2EDuration="26.038356431s" totalImagesPullingTime="22.332184804s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:27.038273093 +0000 UTC m=+26.842696421" watchObservedRunningTime="2026-07-16 21:50:27.038356431 +0000 UTC m=+26.842779759"
Jul 16 21:50:27.043466 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.043430    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:27.147898 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.147863    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:27.605492 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.605434    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:27.821345 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.821301    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:27.821555 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:27.821301    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:27.821555 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:27.821475    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:27.821555 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:27.821520    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:28.749308 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:28.749264    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:28.823887 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:28.823815    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:28.824043 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:28.823943    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:28.849668 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:28.849627    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:29.820541 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:29.820332    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:29.821083 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:29.820392    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:29.821083 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:29.820617    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:29.821083 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:29.820735    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:30.028324 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.028280    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" event={"ID":"aa83eae5-eb8c-440d-98ba-5e8f8b3714b2","Type":"ContainerStarted","Data":"87de98d676785132b9c3f7b5c335f8826cadba90cee0946d5c8de2b97885e614"}
Jul 16 21:50:30.028525 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.028508    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:30.030140 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.030112    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b" containerID="d1dc695d25cad0222f487fc400b11d01aaf4efe988ebfad7a5b46167fcef8387" exitCode=0
Jul 16 21:50:30.030267 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.030150    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" event={"ID":"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b","Type":"ContainerDied","Data":"d1dc695d25cad0222f487fc400b11d01aaf4efe988ebfad7a5b46167fcef8387"}
Jul 16 21:50:30.045527 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.045498    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:30.057571 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.057516    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb" podStartSLOduration=8.314190044 podStartE2EDuration="29.057503381s" totalImagesPullingTime="20.743313337s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:30.056382805 +0000 UTC m=+29.860806142" watchObservedRunningTime="2026-07-16 21:50:30.057503381 +0000 UTC m=+29.861926707"
Jul 16 21:50:30.465146 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.465089    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:30.562351 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.562291    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:30.823966 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:30.823947    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:30.824743 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:30.824074    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:31.034888 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.034667    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b" containerID="a22db19403fde489751a29f8892358dae0d396bc7f4d2f6a07f58b720dc4ce84" exitCode=0
Jul 16 21:50:31.035146 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.034721    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" event={"ID":"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b","Type":"ContainerDied","Data":"a22db19403fde489751a29f8892358dae0d396bc7f4d2f6a07f58b720dc4ce84"}
Jul 16 21:50:31.035394 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.035366    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:31.035471 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.035436    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:31.052547 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.052519    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:50:31.122276 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.122187    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dhp2x"]
Jul 16 21:50:31.122415 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.122334    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:31.122465 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:31.122424    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:31.122822 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.122801    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5wx6j"]
Jul 16 21:50:31.122893 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.122882    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:31.122984 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:31.122962    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:31.124190 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.124156    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9sj7"]
Jul 16 21:50:31.124302 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:31.124260    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:31.124380 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:31.124350    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:32.823527 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:32.823492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:32.823899 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:32.823492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:32.823899 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:32.823608    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:32.823899 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:32.823492    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:32.823899 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:32.823681    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:32.823899 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:32.823748    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:33.042980 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:33.042907    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b" containerID="0d28732fd338dcce45798caf618a286c51e225754e2b1e17886b1222c5b263e4" exitCode=0
Jul 16 21:50:33.043167 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:33.043140    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" event={"ID":"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b","Type":"ContainerDied","Data":"0d28732fd338dcce45798caf618a286c51e225754e2b1e17886b1222c5b263e4"}
Jul 16 21:50:34.389327 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:34.389283    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:34.389881 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.389471    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:34.389881 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.389564    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs podName:3b5ea743-a8b0-4e62-9217-8215b918bee1 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.389539543 +0000 UTC m=+66.193962846 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs") pod "network-metrics-daemon-l9sj7" (UID: "3b5ea743-a8b0-4e62-9217-8215b918bee1") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 16 21:50:34.490201 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:34.490163    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:34.490403 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:34.490232    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:34.490403 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.490341    2585 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:34.490512 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.490410    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret podName:00df8ce7-3aa0-4336-a467-766ba77730b8 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.490390193 +0000 UTC m=+66.294813497 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret") pod "global-pull-secret-syncer-5wx6j" (UID: "00df8ce7-3aa0-4336-a467-766ba77730b8") : object "kube-system"/"original-pull-secret" not registered
Jul 16 21:50:34.490512 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.490353    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 16 21:50:34.490512 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.490467    2585 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 16 21:50:34.490512 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.490487    2585 projected.go:196] Error preparing data for projected volume kube-api-access-rc7hz for pod openshift-network-diagnostics/network-check-target-dhp2x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:34.490725 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.490547    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz podName:4a203a4e-09f9-4ede-927a-38ccaad319b9 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.490528207 +0000 UTC m=+66.294951513 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-rc7hz" (UniqueName: "kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz") pod "network-check-target-dhp2x" (UID: "4a203a4e-09f9-4ede-927a-38ccaad319b9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 16 21:50:34.821241 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:34.821208    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:34.821408 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:34.821207    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:34.821408 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.821336    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:34.821408 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:34.821208    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:34.821577 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.821410    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:34.824531 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:34.821875    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:34.842172 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:34.842129    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:36.226660 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:36.226628    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:36.821116 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:36.821063    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:36.821310 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:36.821064    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:36.821310 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:36.821223    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:36.821310 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:36.821224    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-dhp2x" podUID="4a203a4e-09f9-4ede-927a-38ccaad319b9"
Jul 16 21:50:36.821434 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:36.821322    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-l9sj7" podUID="3b5ea743-a8b0-4e62-9217-8215b918bee1"
Jul 16 21:50:36.821434 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:36.821404    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-5wx6j" podUID="00df8ce7-3aa0-4336-a467-766ba77730b8"
Jul 16 21:50:37.547265 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.547239    2585 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-202.ec2.internal" event="NodeReady"
Jul 16 21:50:37.547698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.547365    2585 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 16 21:50:37.606737 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.606702    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-rpn85"]
Jul 16 21:50:37.609577 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.609559    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.611492 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.611468    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-w7rxp"]
Jul 16 21:50:37.614773 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.614750    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w7rxp"
Jul 16 21:50:37.615517 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.615498    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n6rt\""
Jul 16 21:50:37.615957 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.615919    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 16 21:50:37.616945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.616860    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 16 21:50:37.617043 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.616863    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-pz5gf\""
Jul 16 21:50:37.617372 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.617353    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 16 21:50:37.617482 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.617407    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 16 21:50:37.617482 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.617447    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 16 21:50:37.624623 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.624588    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rpn85"]
Jul 16 21:50:37.627908 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.627881    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w7rxp"]
Jul 16 21:50:37.713852 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.713816    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8j5h\" (UniqueName: \"kubernetes.io/projected/665fbdf6-ab18-40f5-9bc0-5ef816e90bc0-kube-api-access-z8j5h\") pod \"ingress-canary-w7rxp\" (UID: \"665fbdf6-ab18-40f5-9bc0-5ef816e90bc0\") " pod="openshift-ingress-canary/ingress-canary-w7rxp"
Jul 16 21:50:37.714055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.713887    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/665fbdf6-ab18-40f5-9bc0-5ef816e90bc0-cert\") pod \"ingress-canary-w7rxp\" (UID: \"665fbdf6-ab18-40f5-9bc0-5ef816e90bc0\") " pod="openshift-ingress-canary/ingress-canary-w7rxp"
Jul 16 21:50:37.714055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.713981    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90027345-b94e-49c5-8d72-0bff02dcdf87-config-volume\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.714055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.714013    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.714055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.714033    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb6k8\" (UniqueName: \"kubernetes.io/projected/90027345-b94e-49c5-8d72-0bff02dcdf87-kube-api-access-tb6k8\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.714206 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.714062    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90027345-b94e-49c5-8d72-0bff02dcdf87-tmp-dir\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.787237 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.787198    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:37.815084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.814987    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90027345-b94e-49c5-8d72-0bff02dcdf87-config-volume\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.815084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.815042    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.815084 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.815073    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb6k8\" (UniqueName: \"kubernetes.io/projected/90027345-b94e-49c5-8d72-0bff02dcdf87-kube-api-access-tb6k8\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.815369 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.815150    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90027345-b94e-49c5-8d72-0bff02dcdf87-tmp-dir\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.815369 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.815179    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8j5h\" (UniqueName: \"kubernetes.io/projected/665fbdf6-ab18-40f5-9bc0-5ef816e90bc0-kube-api-access-z8j5h\") pod \"ingress-canary-w7rxp\" (UID: \"665fbdf6-ab18-40f5-9bc0-5ef816e90bc0\") " pod="openshift-ingress-canary/ingress-canary-w7rxp"
Jul 16 21:50:37.815369 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:37.815224    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:37.815369 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:37.815306    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls podName:90027345-b94e-49c5-8d72-0bff02dcdf87 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:38.315281249 +0000 UTC m=+38.119704574 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls") pod "dns-default-rpn85" (UID: "90027345-b94e-49c5-8d72-0bff02dcdf87") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:37.815369 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.815236    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/665fbdf6-ab18-40f5-9bc0-5ef816e90bc0-cert\") pod \"ingress-canary-w7rxp\" (UID: \"665fbdf6-ab18-40f5-9bc0-5ef816e90bc0\") " pod="openshift-ingress-canary/ingress-canary-w7rxp"
Jul 16 21:50:37.815767 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.815743    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/90027345-b94e-49c5-8d72-0bff02dcdf87-tmp-dir\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.816192 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.815999    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/90027345-b94e-49c5-8d72-0bff02dcdf87-config-volume\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.819787 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.819765    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/665fbdf6-ab18-40f5-9bc0-5ef816e90bc0-cert\") pod \"ingress-canary-w7rxp\" (UID: \"665fbdf6-ab18-40f5-9bc0-5ef816e90bc0\") " pod="openshift-ingress-canary/ingress-canary-w7rxp"
Jul 16 21:50:37.825163 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.825132    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb6k8\" (UniqueName: \"kubernetes.io/projected/90027345-b94e-49c5-8d72-0bff02dcdf87-kube-api-access-tb6k8\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:37.825343 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.825324    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8j5h\" (UniqueName: \"kubernetes.io/projected/665fbdf6-ab18-40f5-9bc0-5ef816e90bc0-kube-api-access-z8j5h\") pod \"ingress-canary-w7rxp\" (UID: \"665fbdf6-ab18-40f5-9bc0-5ef816e90bc0\") " pod="openshift-ingress-canary/ingress-canary-w7rxp"
Jul 16 21:50:37.888032 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.887989    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:37.926064 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:37.926016    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-w7rxp"
Jul 16 21:50:38.059396 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.059359    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b" containerID="0949dc4a1631fe93cf2460d6256fd210b7db831cc028e79631c40c6245d83301" exitCode=0
Jul 16 21:50:38.059588 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.059422    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" event={"ID":"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b","Type":"ContainerDied","Data":"0949dc4a1631fe93cf2460d6256fd210b7db831cc028e79631c40c6245d83301"}
Jul 16 21:50:38.068712 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.068648    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-w7rxp"]
Jul 16 21:50:38.071854 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:38.071824    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod665fbdf6_ab18_40f5_9bc0_5ef816e90bc0.slice/crio-31da362515e3e8ce5bc34a86eaf97caff68edf3949833c9d14f246dba652910a WatchSource:0}: Error finding container 31da362515e3e8ce5bc34a86eaf97caff68edf3949833c9d14f246dba652910a: Status 404 returned error can't find the container with id 31da362515e3e8ce5bc34a86eaf97caff68edf3949833c9d14f246dba652910a
Jul 16 21:50:38.320048 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.319947    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:38.320225 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:38.320118    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:38.320225 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:38.320188    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls podName:90027345-b94e-49c5-8d72-0bff02dcdf87 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:39.32016879 +0000 UTC m=+39.124592095 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls") pod "dns-default-rpn85" (UID: "90027345-b94e-49c5-8d72-0bff02dcdf87") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:38.677420 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.677336    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:38.772354 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.772316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:38.824869 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.824840    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:50:38.825085 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.824885    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:50:38.825085 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.824952    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:50:38.827714 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.827686    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:50:38.827714 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.827689    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 16 21:50:38.827904 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.827735    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 16 21:50:38.828701 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.828678    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqtjl\""
Jul 16 21:50:38.828818 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.828742    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:50:38.829029 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:38.829009    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xshkd\""
Jul 16 21:50:39.063240 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:39.063202    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w7rxp" event={"ID":"665fbdf6-ab18-40f5-9bc0-5ef816e90bc0","Type":"ContainerStarted","Data":"31da362515e3e8ce5bc34a86eaf97caff68edf3949833c9d14f246dba652910a"}
Jul 16 21:50:39.066873 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:39.066839    2585 generic.go:356] "Generic (PLEG): container finished" podID="f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b" containerID="42544c91773ca2adc672d0d03f7184ee2dc3349603b4794025c1e2e5e84e2b52" exitCode=0
Jul 16 21:50:39.067054 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:39.066898    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" event={"ID":"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b","Type":"ContainerDied","Data":"42544c91773ca2adc672d0d03f7184ee2dc3349603b4794025c1e2e5e84e2b52"}
Jul 16 21:50:39.328133 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:39.328020    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:39.328309 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:39.328181    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:39.328309 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:39.328270    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls podName:90027345-b94e-49c5-8d72-0bff02dcdf87 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:41.328244431 +0000 UTC m=+41.132667737 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls") pod "dns-default-rpn85" (UID: "90027345-b94e-49c5-8d72-0bff02dcdf87") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:40.077284 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:40.077246    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" event={"ID":"f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b","Type":"ContainerStarted","Data":"0ef9801b9e0fb83558904c8d9568b8479459d12dbb6cbe7e07362b1a5fa8865b"}
Jul 16 21:50:40.117893 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:40.117825    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-xwr8m" podStartSLOduration=8.668741334 podStartE2EDuration="39.117804544s" totalImagesPullingTime="30.44906321s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:40.116978747 +0000 UTC m=+39.921402074" watchObservedRunningTime="2026-07-16 21:50:40.117804544 +0000 UTC m=+39.922227870"
Jul 16 21:50:40.256905 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:40.256856    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:40.343012 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:40.342935    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:41.081193 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:41.081154    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-w7rxp" event={"ID":"665fbdf6-ab18-40f5-9bc0-5ef816e90bc0","Type":"ContainerStarted","Data":"7a648dfdfb5a658405ab039e75055851332a439ca88d1b2076c7010288207f4c"}
Jul 16 21:50:41.094026 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:41.093988    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:41.102318 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:41.102269    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-w7rxp" podStartSLOduration=1.580221839 podStartE2EDuration="4.102251137s" totalImagesPullingTime="2.522029298s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:50:41.10223034 +0000 UTC m=+40.906653665" watchObservedRunningTime="2026-07-16 21:50:41.102251137 +0000 UTC m=+40.906674513"
Jul 16 21:50:41.193702 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:41.193662    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:41.345792 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:41.345681    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:41.345954 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:41.345843    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:41.345954 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:41.345925    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls podName:90027345-b94e-49c5-8d72-0bff02dcdf87 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:45.345908013 +0000 UTC m=+45.150331320 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls") pod "dns-default-rpn85" (UID: "90027345-b94e-49c5-8d72-0bff02dcdf87") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:45.373865 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:45.373822    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:45.374286 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:45.373983    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:45.374286 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:45.374049    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls podName:90027345-b94e-49c5-8d72-0bff02dcdf87 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:53.374033431 +0000 UTC m=+53.178456735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls") pod "dns-default-rpn85" (UID: "90027345-b94e-49c5-8d72-0bff02dcdf87") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:48.023192 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:48.023151    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:48.285922 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:48.285828    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:53.433274 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:53.433227    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:50:53.433763 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:53.433385    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:50:53.433763 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:53.433458    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls podName:90027345-b94e-49c5-8d72-0bff02dcdf87 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:09.433437614 +0000 UTC m=+69.237860918 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls") pod "dns-default-rpn85" (UID: "90027345-b94e-49c5-8d72-0bff02dcdf87") : secret "dns-default-metrics-tls" not found
Jul 16 21:50:58.118542 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.118507    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd"]
Jul 16 21:50:58.120419 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.120402    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd"
Jul 16 21:50:58.123082 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.123051    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:58.123750 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.123721    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-tzz4j\""
Jul 16 21:50:58.123750 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.123723    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:58.128343 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.128315    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-6c9998cdd8-f8hkr"]
Jul 16 21:50:58.130252 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.130234    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-555d9d7885-4bfcb"]
Jul 16 21:50:58.130367 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.130351    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.131907 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.131886    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd"]
Jul 16 21:50:58.132019 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.131977    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.132632 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.132609    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 16 21:50:58.133168 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.133148    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nmm9q\""
Jul 16 21:50:58.133888 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.133867    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 16 21:50:58.133986 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.133901    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 16 21:50:58.134962 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.134945    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 16 21:50:58.135066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.134947    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 16 21:50:58.135066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.134997    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-67glt\""
Jul 16 21:50:58.135185 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.134947    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 16 21:50:58.135309 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.135292    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 16 21:50:58.142217 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.142191    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 16 21:50:58.142813 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.142792    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 16 21:50:58.148521 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.148497    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-4bfcb"]
Jul 16 21:50:58.155867 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.155837    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c9998cdd8-f8hkr"]
Jul 16 21:50:58.167460 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5960e6-9c85-4d5a-947e-c4075b287904-service-ca-bundle\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.167460 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167467    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.167710 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-bound-sa-token\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.167710 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167567    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5960e6-9c85-4d5a-947e-c4075b287904-serving-cert\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.167710 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167596    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzn4q\" (UniqueName: \"kubernetes.io/projected/e8f0d57d-ab86-4bcd-8587-5f8582652a6c-kube-api-access-jzn4q\") pod \"volume-data-source-validator-78cf4c6f6-jttvd\" (UID: \"e8f0d57d-ab86-4bcd-8587-5f8582652a6c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd"
Jul 16 21:50:58.167710 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167614    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-trusted-ca\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.167710 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-installation-pull-secrets\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.167710 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167686    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l96vf\" (UniqueName: \"kubernetes.io/projected/2a5960e6-9c85-4d5a-947e-c4075b287904-kube-api-access-l96vf\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.167903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167719    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-certificates\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.167903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167745    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t29xn\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-kube-api-access-t29xn\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.167903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167774    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a5960e6-9c85-4d5a-947e-c4075b287904-snapshots\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.167903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167807    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a5960e6-9c85-4d5a-947e-c4075b287904-tmp\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.167903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167822    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5960e6-9c85-4d5a-947e-c4075b287904-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.167903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167853    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-image-registry-private-configuration\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.167903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.167869    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22bc564-2ae4-44fa-ad34-4a3761415500-ca-trust-extracted\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.236740 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.236704    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"]
Jul 16 21:50:58.238526 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.238508    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.241197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.241164    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 16 21:50:58.241493 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.241478    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 16 21:50:58.241538 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.241519    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 16 21:50:58.242228 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.242212    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 16 21:50:58.242538 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.242522    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zkdw2\""
Jul 16 21:50:58.250346 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.250320    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk"]
Jul 16 21:50:58.252962 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.252939    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5jwjh"]
Jul 16 21:50:58.253092 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.253076    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk"
Jul 16 21:50:58.254498 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.254483    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.255396 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.255377    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-h4z9t\""
Jul 16 21:50:58.259381 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.259362    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 16 21:50:58.259492 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.259393    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 16 21:50:58.260197 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.260184    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-8hj4z\""
Jul 16 21:50:58.263156 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.263134    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 16 21:50:58.264591 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.264571    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 16 21:50:58.268176 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268149    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22bc564-2ae4-44fa-ad34-4a3761415500-ca-trust-extracted\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.268296 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268194    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a5960e6-9c85-4d5a-947e-c4075b287904-snapshots\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.268356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a5960e6-9c85-4d5a-947e-c4075b287904-tmp\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.268356 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268339    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86r79\" (UniqueName: \"kubernetes.io/projected/8999468a-beac-4f06-bfba-b24e3d377eaa-kube-api-access-86r79\") pod \"network-check-source-7bc5b6bf85-95wpk\" (UID: \"8999468a-beac-4f06-bfba-b24e3d377eaa\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk"
Jul 16 21:50:58.268466 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268371    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5960e6-9c85-4d5a-947e-c4075b287904-service-ca-bundle\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.268466 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268401    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z8t7w\" (UniqueName: \"kubernetes.io/projected/aa88b577-be62-48bb-bf4d-2eadb3b2d857-kube-api-access-z8t7w\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.268466 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268429    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.268466 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268446    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-bound-sa-token\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268467    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-trusted-ca\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268491    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5960e6-9c85-4d5a-947e-c4075b287904-serving-cert\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268512    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzn4q\" (UniqueName: \"kubernetes.io/projected/e8f0d57d-ab86-4bcd-8587-5f8582652a6c-kube-api-access-jzn4q\") pod \"volume-data-source-validator-78cf4c6f6-jttvd\" (UID: \"e8f0d57d-ab86-4bcd-8587-5f8582652a6c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd"
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268528    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-installation-pull-secrets\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.268541    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268549    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l96vf\" (UniqueName: \"kubernetes.io/projected/2a5960e6-9c85-4d5a-947e-c4075b287904-kube-api-access-l96vf\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.268558    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c9998cdd8-f8hkr: secret "image-registry-tls" not found
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268576    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-certificates\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268605    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv7dp\" (UniqueName: \"kubernetes.io/projected/1defe86c-f9b6-4b89-b263-3f8d84c15935-kube-api-access-pv7dp\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.268653 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.268624    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls podName:d22bc564-2ae4-44fa-ad34-4a3761415500 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:58.768603598 +0000 UTC m=+58.573026908 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls") pod "image-registry-6c9998cdd8-f8hkr" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500") : secret "image-registry-tls" not found
Jul 16 21:50:58.269121 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268662    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5960e6-9c85-4d5a-947e-c4075b287904-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.269121 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268697    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-image-registry-private-configuration\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.269121 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268748    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.269121 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268777    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa88b577-be62-48bb-bf4d-2eadb3b2d857-config\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.269121 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268819    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2a5960e6-9c85-4d5a-947e-c4075b287904-tmp\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.269121 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268890    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/2a5960e6-9c85-4d5a-947e-c4075b287904-snapshots\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.269121 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.268549    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22bc564-2ae4-44fa-ad34-4a3761415500-ca-trust-extracted\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.269121 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.269026    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5960e6-9c85-4d5a-947e-c4075b287904-service-ca-bundle\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.269573 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.269546    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-certificates\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.269717 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.269684    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1defe86c-f9b6-4b89-b263-3f8d84c15935-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.269845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.269747    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa88b577-be62-48bb-bf4d-2eadb3b2d857-serving-cert\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.269845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.269787    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t29xn\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-kube-api-access-t29xn\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.269845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.269834    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa88b577-be62-48bb-bf4d-2eadb3b2d857-trusted-ca\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.270013 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.269898    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/2a5960e6-9c85-4d5a-947e-c4075b287904-trusted-ca-bundle\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.270071 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.270019    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-trusted-ca\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.272770 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.272745    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 16 21:50:58.272926 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.272902    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-installation-pull-secrets\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.273023 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.272924    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/2a5960e6-9c85-4d5a-947e-c4075b287904-serving-cert\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.273023 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.273010    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-image-registry-private-configuration\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.297285 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.297252    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk"]
Jul 16 21:50:58.305394 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.305362    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-bound-sa-token\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.305707 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.305689    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzn4q\" (UniqueName: \"kubernetes.io/projected/e8f0d57d-ab86-4bcd-8587-5f8582652a6c-kube-api-access-jzn4q\") pod \"volume-data-source-validator-78cf4c6f6-jttvd\" (UID: \"e8f0d57d-ab86-4bcd-8587-5f8582652a6c\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd"
Jul 16 21:50:58.306486 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.306467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l96vf\" (UniqueName: \"kubernetes.io/projected/2a5960e6-9c85-4d5a-947e-c4075b287904-kube-api-access-l96vf\") pod \"insights-operator-555d9d7885-4bfcb\" (UID: \"2a5960e6-9c85-4d5a-947e-c4075b287904\") " pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.307232 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.307215    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t29xn\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-kube-api-access-t29xn\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.318029 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.317988    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5jwjh"]
Jul 16 21:50:58.330011 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.329975    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"]
Jul 16 21:50:58.370745 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.370635    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86r79\" (UniqueName: \"kubernetes.io/projected/8999468a-beac-4f06-bfba-b24e3d377eaa-kube-api-access-86r79\") pod \"network-check-source-7bc5b6bf85-95wpk\" (UID: \"8999468a-beac-4f06-bfba-b24e3d377eaa\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk"
Jul 16 21:50:58.370745 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.370695    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z8t7w\" (UniqueName: \"kubernetes.io/projected/aa88b577-be62-48bb-bf4d-2eadb3b2d857-kube-api-access-z8t7w\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.370745 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.370739    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv7dp\" (UniqueName: \"kubernetes.io/projected/1defe86c-f9b6-4b89-b263-3f8d84c15935-kube-api-access-pv7dp\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.371007 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.370778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.371007 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.370801    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa88b577-be62-48bb-bf4d-2eadb3b2d857-config\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.371007 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.370832    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1defe86c-f9b6-4b89-b263-3f8d84c15935-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.371007 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.370900    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:58.371007 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.370908    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa88b577-be62-48bb-bf4d-2eadb3b2d857-serving-cert\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.371007 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.370971    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls podName:1defe86c-f9b6-4b89-b263-3f8d84c15935 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:58.870949525 +0000 UTC m=+58.675372830 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-grzb8" (UID: "1defe86c-f9b6-4b89-b263-3f8d84c15935") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:58.371007 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.370992    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa88b577-be62-48bb-bf4d-2eadb3b2d857-trusted-ca\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.371622 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.371584    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/aa88b577-be62-48bb-bf4d-2eadb3b2d857-config\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.371622 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.371622    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/1defe86c-f9b6-4b89-b263-3f8d84c15935-telemetry-config\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.371782 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.371673    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/aa88b577-be62-48bb-bf4d-2eadb3b2d857-trusted-ca\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.373341 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.373324    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/aa88b577-be62-48bb-bf4d-2eadb3b2d857-serving-cert\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.383423 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.383394    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-86r79\" (UniqueName: \"kubernetes.io/projected/8999468a-beac-4f06-bfba-b24e3d377eaa-kube-api-access-86r79\") pod \"network-check-source-7bc5b6bf85-95wpk\" (UID: \"8999468a-beac-4f06-bfba-b24e3d377eaa\") " pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk"
Jul 16 21:50:58.388719 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.388696    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z8t7w\" (UniqueName: \"kubernetes.io/projected/aa88b577-be62-48bb-bf4d-2eadb3b2d857-kube-api-access-z8t7w\") pod \"console-operator-565b7b569d-5jwjh\" (UID: \"aa88b577-be62-48bb-bf4d-2eadb3b2d857\") " pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.390478 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.390459    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv7dp\" (UniqueName: \"kubernetes.io/projected/1defe86c-f9b6-4b89-b263-3f8d84c15935-kube-api-access-pv7dp\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.430329 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.430294    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd"
Jul 16 21:50:58.447299 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.447272    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-555d9d7885-4bfcb"
Jul 16 21:50:58.561873 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.561833    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk"
Jul 16 21:50:58.567217 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.567186    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:50:58.589897 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.589861    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd"]
Jul 16 21:50:58.595226 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:58.595197    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8f0d57d_ab86_4bcd_8587_5f8582652a6c.slice/crio-544113d3bdae65bb07a50a084b34ce5233a5c864d11c673bbc3e07ebf520e3f2 WatchSource:0}: Error finding container 544113d3bdae65bb07a50a084b34ce5233a5c864d11c673bbc3e07ebf520e3f2: Status 404 returned error can't find the container with id 544113d3bdae65bb07a50a084b34ce5233a5c864d11c673bbc3e07ebf520e3f2
Jul 16 21:50:58.613893 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.613802    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-555d9d7885-4bfcb"]
Jul 16 21:50:58.618162 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:58.618131    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a5960e6_9c85_4d5a_947e_c4075b287904.slice/crio-652b85b469170fc14cb84877d5adce5eaa94789f3721c209814047de2afa0aee WatchSource:0}: Error finding container 652b85b469170fc14cb84877d5adce5eaa94789f3721c209814047de2afa0aee: Status 404 returned error can't find the container with id 652b85b469170fc14cb84877d5adce5eaa94789f3721c209814047de2afa0aee
Jul 16 21:50:58.667568 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.667523    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:58.735738 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.735497    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk"]
Jul 16 21:50:58.738164 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:58.738140    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8999468a_beac_4f06_bfba_b24e3d377eaa.slice/crio-0b17ba9b38916b22c18b1fd4715bbbaadc9794118d49ee05bd1934352968b6d7 WatchSource:0}: Error finding container 0b17ba9b38916b22c18b1fd4715bbbaadc9794118d49ee05bd1934352968b6d7: Status 404 returned error can't find the container with id 0b17ba9b38916b22c18b1fd4715bbbaadc9794118d49ee05bd1934352968b6d7
Jul 16 21:50:58.753564 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.753534    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-565b7b569d-5jwjh"]
Jul 16 21:50:58.756829 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:50:58.756805    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa88b577_be62_48bb_bf4d_2eadb3b2d857.slice/crio-e0b070f30aaf7a809ede4ac1ce38d5d552574696af7d2aee34ee976ff12defd2 WatchSource:0}: Error finding container e0b070f30aaf7a809ede4ac1ce38d5d552574696af7d2aee34ee976ff12defd2: Status 404 returned error can't find the container with id e0b070f30aaf7a809ede4ac1ce38d5d552574696af7d2aee34ee976ff12defd2
Jul 16 21:50:58.757500 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.757460    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:50:58.774624 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.774579    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:58.774771 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.774729    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:58.774771 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.774746    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c9998cdd8-f8hkr: secret "image-registry-tls" not found
Jul 16 21:50:58.774868 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.774797    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls podName:d22bc564-2ae4-44fa-ad34-4a3761415500 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:59.77478348 +0000 UTC m=+59.579206787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls") pod "image-registry-6c9998cdd8-f8hkr" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500") : secret "image-registry-tls" not found
Jul 16 21:50:58.875282 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:58.875190    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:58.875433 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.875310    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:58.875433 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:58.875364    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls podName:1defe86c-f9b6-4b89-b263-3f8d84c15935 nodeName:}" failed. No retries permitted until 2026-07-16 21:50:59.875350328 +0000 UTC m=+59.679773632 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-grzb8" (UID: "1defe86c-f9b6-4b89-b263-3f8d84c15935") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:59.122808 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:59.122770    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" event={"ID":"aa88b577-be62-48bb-bf4d-2eadb3b2d857","Type":"ContainerStarted","Data":"e0b070f30aaf7a809ede4ac1ce38d5d552574696af7d2aee34ee976ff12defd2"}
Jul 16 21:50:59.124301 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:59.124261    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-4bfcb" event={"ID":"2a5960e6-9c85-4d5a-947e-c4075b287904","Type":"ContainerStarted","Data":"652b85b469170fc14cb84877d5adce5eaa94789f3721c209814047de2afa0aee"}
Jul 16 21:50:59.125697 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:59.125617    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk" event={"ID":"8999468a-beac-4f06-bfba-b24e3d377eaa","Type":"ContainerStarted","Data":"0b17ba9b38916b22c18b1fd4715bbbaadc9794118d49ee05bd1934352968b6d7"}
Jul 16 21:50:59.126982 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:59.126935    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd" event={"ID":"e8f0d57d-ab86-4bcd-8587-5f8582652a6c","Type":"ContainerStarted","Data":"544113d3bdae65bb07a50a084b34ce5233a5c864d11c673bbc3e07ebf520e3f2"}
Jul 16 21:50:59.784635 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:59.783919    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:50:59.784635 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:59.784092    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:50:59.784635 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:59.784129    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c9998cdd8-f8hkr: secret "image-registry-tls" not found
Jul 16 21:50:59.784635 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:59.784199    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls podName:d22bc564-2ae4-44fa-ad34-4a3761415500 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:01.784173663 +0000 UTC m=+61.588596984 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls") pod "image-registry-6c9998cdd8-f8hkr" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500") : secret "image-registry-tls" not found
Jul 16 21:50:59.886055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:50:59.885413    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:50:59.886055 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:59.885623    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:50:59.886055 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:50:59.885688    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls podName:1defe86c-f9b6-4b89-b263-3f8d84c15935 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:01.8856694 +0000 UTC m=+61.690092721 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-grzb8" (UID: "1defe86c-f9b6-4b89-b263-3f8d84c15935") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:51:00.671758 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:00.671721    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:00.768279 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:00.768233    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:01.683403 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:01.683268    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:01.766915 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:01.766871    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:01.802144 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:01.802087    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:01.802344 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:01.802300    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:51:01.802344 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:01.802325    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c9998cdd8-f8hkr: secret "image-registry-tls" not found
Jul 16 21:51:01.802436 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:01.802390    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls podName:d22bc564-2ae4-44fa-ad34-4a3761415500 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.802373685 +0000 UTC m=+65.606796989 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls") pod "image-registry-6c9998cdd8-f8hkr" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500") : secret "image-registry-tls" not found
Jul 16 21:51:01.902808 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:01.902765    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:51:01.903018 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:01.902955    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:51:01.903078 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:01.903038    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls podName:1defe86c-f9b6-4b89-b263-3f8d84c15935 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.903015486 +0000 UTC m=+65.707438790 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-grzb8" (UID: "1defe86c-f9b6-4b89-b263-3f8d84c15935") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:51:01.946521 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:01.946420    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:02.027923 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.027889    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:02.391878 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.391813    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:02.476081 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.476041    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:02.667385 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.667352    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:02.801204 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.801164    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"]
Jul 16 21:51:02.804782 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.804738    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:02.807614 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.807385    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-rtcl7\""
Jul 16 21:51:02.807614 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.807544    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 16 21:51:02.808389 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.808081    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 16 21:51:02.816784 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.816744    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"]
Jul 16 21:51:02.830888 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.830840    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:02.913237 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.913146    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:02.913237 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.913196    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9469eba-c73f-4135-aa80-f12cf16ee555-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:02.935194 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:02.935162    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:03.014220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.014173    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:03.014426 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.014231    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9469eba-c73f-4135-aa80-f12cf16ee555-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:03.014483 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:03.014427    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:03.014529 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:03.014516    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert podName:c9469eba-c73f-4135-aa80-f12cf16ee555 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:03.514492312 +0000 UTC m=+63.318915639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-l262c" (UID: "c9469eba-c73f-4135-aa80-f12cf16ee555") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:03.014842 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.014825    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/c9469eba-c73f-4135-aa80-f12cf16ee555-nginx-conf\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:03.054885 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.054852    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-hnjgb"
Jul 16 21:51:03.141022 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.140994    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/0.log"
Jul 16 21:51:03.141235 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.141037    2585 generic.go:356] "Generic (PLEG): container finished" podID="aa88b577-be62-48bb-bf4d-2eadb3b2d857" containerID="24e82a6a477e1accaa0bfc873d2713716fa51e7d9d5ed6f2696975fec5640699" exitCode=255
Jul 16 21:51:03.141235 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.141079    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" event={"ID":"aa88b577-be62-48bb-bf4d-2eadb3b2d857","Type":"ContainerDied","Data":"24e82a6a477e1accaa0bfc873d2713716fa51e7d9d5ed6f2696975fec5640699"}
Jul 16 21:51:03.141427 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.141396    2585 scope.go:117] "RemoveContainer" containerID="24e82a6a477e1accaa0bfc873d2713716fa51e7d9d5ed6f2696975fec5640699"
Jul 16 21:51:03.142822 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.142790    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-4bfcb" event={"ID":"2a5960e6-9c85-4d5a-947e-c4075b287904","Type":"ContainerStarted","Data":"38c46b18011f5c0159dee2cd16a3d277e84ca334df29e4a1cfbd35d03c7f5909"}
Jul 16 21:51:03.144446 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.144416    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk" event={"ID":"8999468a-beac-4f06-bfba-b24e3d377eaa","Type":"ContainerStarted","Data":"f28314f87b893d36d31caee8653c2ad575f364ec1e1306385689a24f80adbece"}
Jul 16 21:51:03.146138 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.146110    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd" event={"ID":"e8f0d57d-ab86-4bcd-8587-5f8582652a6c","Type":"ContainerStarted","Data":"86957288fc9c3a927a089dd727b926de247a27731827d569bc220b28a64936c1"}
Jul 16 21:51:03.187964 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.187912    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-555d9d7885-4bfcb" podStartSLOduration=1.128893475 podStartE2EDuration="5.187893847s" totalImagesPullingTime="4.059000372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:03.187381463 +0000 UTC m=+62.991804794" watchObservedRunningTime="2026-07-16 21:51:03.187893847 +0000 UTC m=+62.992317172"
Jul 16 21:51:03.226552 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.226387    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78cf4c6f6-jttvd" podStartSLOduration=1.144997222 podStartE2EDuration="5.226365474s" totalImagesPullingTime="4.081368252s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:03.225016824 +0000 UTC m=+63.029440151" watchObservedRunningTime="2026-07-16 21:51:03.226365474 +0000 UTC m=+63.030788800"
Jul 16 21:51:03.250021 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.249953    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7bc5b6bf85-95wpk" podStartSLOduration=1.262265733 podStartE2EDuration="5.249927122s" totalImagesPullingTime="3.987661389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:03.249846282 +0000 UTC m=+63.054269610" watchObservedRunningTime="2026-07-16 21:51:03.249927122 +0000 UTC m=+63.054350450"
Jul 16 21:51:03.277873 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.277834    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:03.384250 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.384214    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:03.518941 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.518889    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:03.519152 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:03.519035    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:03.519152 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:03.519115    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert podName:c9469eba-c73f-4135-aa80-f12cf16ee555 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.519085415 +0000 UTC m=+64.323508718 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-l262c" (UID: "c9469eba-c73f-4135-aa80-f12cf16ee555") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:03.978934 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:03.978830    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:04.072949 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.072905    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-mfqnh"]
Jul 16 21:51:04.075966 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.075938    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.078957 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.078933    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 16 21:51:04.079285 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.079267    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-8tffc\""
Jul 16 21:51:04.079802 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.079787    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 16 21:51:04.095350 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.095310    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mfqnh"]
Jul 16 21:51:04.125553 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.125513    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf883ff1-b068-427a-b5e7-d69cbf631112-data-volume\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.125775 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.125637    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmtwp\" (UniqueName: \"kubernetes.io/projected/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-api-access-wmtwp\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.125775 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.125726    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.125874 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.125778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf883ff1-b068-427a-b5e7-d69cbf631112-crio-socket\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.125874 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.125799    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.127014 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.126986    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:04.151319 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.151287    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 21:51:04.151776 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.151760    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/0.log"
Jul 16 21:51:04.151827 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.151797    2585 generic.go:356] "Generic (PLEG): container finished" podID="aa88b577-be62-48bb-bf4d-2eadb3b2d857" containerID="bdf7c9d421b8e3b22c62f3e67acdea4fd6a9d51240476c507ae4e6125e8bab28" exitCode=255
Jul 16 21:51:04.151906 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.151891    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" event={"ID":"aa88b577-be62-48bb-bf4d-2eadb3b2d857","Type":"ContainerDied","Data":"bdf7c9d421b8e3b22c62f3e67acdea4fd6a9d51240476c507ae4e6125e8bab28"}
Jul 16 21:51:04.151973 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.151962    2585 scope.go:117] "RemoveContainer" containerID="24e82a6a477e1accaa0bfc873d2713716fa51e7d9d5ed6f2696975fec5640699"
Jul 16 21:51:04.152287 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.152264    2585 scope.go:117] "RemoveContainer" containerID="bdf7c9d421b8e3b22c62f3e67acdea4fd6a9d51240476c507ae4e6125e8bab28"
Jul 16 21:51:04.152489 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.152454    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5jwjh_openshift-console-operator(aa88b577-be62-48bb-bf4d-2eadb3b2d857)\"" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" podUID="aa88b577-be62-48bb-bf4d-2eadb3b2d857"
Jul 16 21:51:04.226220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.226183    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf883ff1-b068-427a-b5e7-d69cbf631112-crio-socket\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.226220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.226226    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.226517 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.226322    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/bf883ff1-b068-427a-b5e7-d69cbf631112-crio-socket\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.226517 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.226356    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:04.226517 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.226415    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.726397808 +0000 UTC m=+64.530821142 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:04.226517 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.226433    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf883ff1-b068-427a-b5e7-d69cbf631112-data-volume\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.226743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.226563    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmtwp\" (UniqueName: \"kubernetes.io/projected/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-api-access-wmtwp\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.226743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.226723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.226852 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.226758    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/bf883ff1-b068-427a-b5e7-d69cbf631112-data-volume\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.226910 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.226857    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:04.226963 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.226921    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:04.726903101 +0000 UTC m=+64.531326425 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:04.247429 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.247389    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmtwp\" (UniqueName: \"kubernetes.io/projected/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-api-access-wmtwp\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.333732 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.333699    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:04.451602 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.451568    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:04.530035 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.529913    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:04.530242 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.530079    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:04.530242 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.530179    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert podName:c9469eba-c73f-4135-aa80-f12cf16ee555 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:06.530161602 +0000 UTC m=+66.334584905 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-l262c" (UID: "c9469eba-c73f-4135-aa80-f12cf16ee555") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:04.731665 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.731626    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.731873 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:04.731680    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:04.731873 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.731758    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:04.731873 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.731786    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:04.731873 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.731831    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.731812135 +0000 UTC m=+65.536235439 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:04.731873 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:04.731847    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:05.731839547 +0000 UTC m=+65.536262851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:05.155955 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:05.155922    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 21:51:05.156380 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:05.156292    2585 scope.go:117] "RemoveContainer" containerID="bdf7c9d421b8e3b22c62f3e67acdea4fd6a9d51240476c507ae4e6125e8bab28"
Jul 16 21:51:05.156465 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.156446    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5jwjh_openshift-console-operator(aa88b577-be62-48bb-bf4d-2eadb3b2d857)\"" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" podUID="aa88b577-be62-48bb-bf4d-2eadb3b2d857"
Jul 16 21:51:05.740268 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:05.740228    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:05.740460 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.740358    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:05.740460 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.740412    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:05.740460 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.740423    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:07.740408202 +0000 UTC m=+67.544831541 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:05.740460 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:05.740362    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:05.740604 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.740465    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:07.740450261 +0000 UTC m=+67.544873568 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:05.841634 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:05.841601    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:05.841818 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.841708    2585 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 16 21:51:05.841818 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.841720    2585 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-6c9998cdd8-f8hkr: secret "image-registry-tls" not found
Jul 16 21:51:05.841818 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.841773    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls podName:d22bc564-2ae4-44fa-ad34-4a3761415500 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:13.841758911 +0000 UTC m=+73.646182215 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls") pod "image-registry-6c9998cdd8-f8hkr" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500") : secret "image-registry-tls" not found
Jul 16 21:51:05.942749 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:05.942705    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:51:05.942918 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.942848    2585 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 16 21:51:05.942918 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:05.942913    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls podName:1defe86c-f9b6-4b89-b263-3f8d84c15935 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:13.942898561 +0000 UTC m=+73.747321864 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-76cf8966f8-grzb8" (UID: "1defe86c-f9b6-4b89-b263-3f8d84c15935") : secret "cluster-monitoring-operator-tls" not found
Jul 16 21:51:05.997201 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:05.997116    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:06.446659 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.446560    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:51:06.449605 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.449581    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 16 21:51:06.457026 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:06.456992    2585 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 16 21:51:06.457154 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:06.457091    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs podName:3b5ea743-a8b0-4e62-9217-8215b918bee1 nodeName:}" failed. No retries permitted until 2026-07-16 21:52:10.457068163 +0000 UTC m=+130.261491468 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs") pod "network-metrics-daemon-l9sj7" (UID: "3b5ea743-a8b0-4e62-9217-8215b918bee1") : secret "metrics-daemon-secret" not found
Jul 16 21:51:06.547181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.547142    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:51:06.547323 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.547270    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:51:06.547323 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.547301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:06.547455 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:06.547438    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:06.547523 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:06.547513    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert podName:c9469eba-c73f-4135-aa80-f12cf16ee555 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:10.54749324 +0000 UTC m=+70.351916558 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-l262c" (UID: "c9469eba-c73f-4135-aa80-f12cf16ee555") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:06.549878 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.549845    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rc7hz\" (UniqueName: \"kubernetes.io/projected/4a203a4e-09f9-4ede-927a-38ccaad319b9-kube-api-access-rc7hz\") pod \"network-check-target-dhp2x\" (UID: \"4a203a4e-09f9-4ede-927a-38ccaad319b9\") " pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:51:06.550409 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.550394    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 16 21:51:06.560534 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.560499    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/00df8ce7-3aa0-4336-a467-766ba77730b8-original-pull-secret\") pod \"global-pull-secret-syncer-5wx6j\" (UID: \"00df8ce7-3aa0-4336-a467-766ba77730b8\") " pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:51:06.737164 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.737135    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-5wx6j"
Jul 16 21:51:06.747416 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.747389    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-nqtjl\""
Jul 16 21:51:06.754907 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.754874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:51:06.884022 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.883989    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-5wx6j"]
Jul 16 21:51:06.886979 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:06.886955    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00df8ce7_3aa0_4336_a467_766ba77730b8.slice/crio-280e8173c0c68d02edffaf27398208ffbfd5ab41c70caa405b63ef08a7ca8fa6 WatchSource:0}: Error finding container 280e8173c0c68d02edffaf27398208ffbfd5ab41c70caa405b63ef08a7ca8fa6: Status 404 returned error can't find the container with id 280e8173c0c68d02edffaf27398208ffbfd5ab41c70caa405b63ef08a7ca8fa6
Jul 16 21:51:06.910868 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.910839    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-dhp2x"]
Jul 16 21:51:06.913274 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:06.913247    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a203a4e_09f9_4ede_927a_38ccaad319b9.slice/crio-587e0fa03702e952d74fcae6105a66d2bacc1846d5d9e8787e3ab1c928fbdfbd WatchSource:0}: Error finding container 587e0fa03702e952d74fcae6105a66d2bacc1846d5d9e8787e3ab1c928fbdfbd: Status 404 returned error can't find the container with id 587e0fa03702e952d74fcae6105a66d2bacc1846d5d9e8787e3ab1c928fbdfbd
Jul 16 21:51:06.970209 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:06.970172    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:07.029823 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.029744    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nj98c_6e9282e2-e6f4-456b-84f7-68872f2c21d0/dns-node-resolver/0.log"
Jul 16 21:51:07.162989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.162946    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dhp2x" event={"ID":"4a203a4e-09f9-4ede-927a-38ccaad319b9","Type":"ContainerStarted","Data":"277a7e5b2f887c444901ab019d429571d69f527f17caed2e43ac47fd788aa8c3"}
Jul 16 21:51:07.163198 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.163008    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-dhp2x" event={"ID":"4a203a4e-09f9-4ede-927a-38ccaad319b9","Type":"ContainerStarted","Data":"587e0fa03702e952d74fcae6105a66d2bacc1846d5d9e8787e3ab1c928fbdfbd"}
Jul 16 21:51:07.163198 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.163129    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:51:07.167836 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.167804    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5wx6j" event={"ID":"00df8ce7-3aa0-4336-a467-766ba77730b8","Type":"ContainerStarted","Data":"280e8173c0c68d02edffaf27398208ffbfd5ab41c70caa405b63ef08a7ca8fa6"}
Jul 16 21:51:07.187474 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.187415    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-dhp2x" podStartSLOduration=66.187399181 podStartE2EDuration="1m6.187399181s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:07.187066395 +0000 UTC m=+66.991489721" watchObservedRunningTime="2026-07-16 21:51:07.187399181 +0000 UTC m=+66.991822537"
Jul 16 21:51:07.205318 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.205282    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:07.361900 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.361806    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:07.515082 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.515041    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:07.619987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.619899    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:07.755905 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.755858    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:07.756226 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:07.756000    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:07.756226 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.756053    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:07.756226 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:07.756075    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:11.756059687 +0000 UTC m=+71.560482990 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:07.756226 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:07.756176    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:07.756489 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:07.756248    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:11.756231851 +0000 UTC m=+71.560655158 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:07.832516 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:07.832489    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jf8qh_24330e43-7607-4cc4-adb6-57d6475d44a9/node-ca/0.log"
Jul 16 21:51:08.407641 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:08.407596    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:08.516970 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:08.516913    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:08.568162 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:08.568127    2585 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:51:08.568362 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:08.568185    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:51:08.568591 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:08.568575    2585 scope.go:117] "RemoveContainer" containerID="bdf7c9d421b8e3b22c62f3e67acdea4fd6a9d51240476c507ae4e6125e8bab28"
Jul 16 21:51:08.568769 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:08.568742    2585 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-565b7b569d-5jwjh_openshift-console-operator(aa88b577-be62-48bb-bf4d-2eadb3b2d857)\"" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" podUID="aa88b577-be62-48bb-bf4d-2eadb3b2d857"
Jul 16 21:51:08.829214 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:08.829176    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w7rxp_665fbdf6-ab18-40f5-9bc0-5ef816e90bc0/serve-healthcheck-canary/0.log"
Jul 16 21:51:09.049609 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:09.049565    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:09.252869 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:09.252829    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:09.448381 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:09.448344    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:09.471360 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:09.471322    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:51:09.471561 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:09.471534    2585 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 16 21:51:09.471635 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:09.471623    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls podName:90027345-b94e-49c5-8d72-0bff02dcdf87 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:41.47160419 +0000 UTC m=+101.276027517 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls") pod "dns-default-rpn85" (UID: "90027345-b94e-49c5-8d72-0bff02dcdf87") : secret "dns-default-metrics-tls" not found
Jul 16 21:51:09.855113 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:09.855061    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:10.041452 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:10.041414    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:10.583488 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:10.583444    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:10.583680 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:10.583613    2585 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 16 21:51:10.583739 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:10.583721    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert podName:c9469eba-c73f-4135-aa80-f12cf16ee555 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:18.583703411 +0000 UTC m=+78.388126716 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert") pod "networking-console-plugin-7bc7489cb5-l262c" (UID: "c9469eba-c73f-4135-aa80-f12cf16ee555") : secret "networking-console-plugin-cert" not found
Jul 16 21:51:10.599565 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:10.599523    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:10.846563 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:10.846477    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:10.946795 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:10.946759    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:11.440832 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:11.440743    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:11.516616 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:11.516574    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:11.642267 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:11.642234    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:11.794213 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:11.794172    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:11.794364 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:11.794241    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:11.794364 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:11.794302    2585 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 16 21:51:11.794432 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:11.794368    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:19.794351498 +0000 UTC m=+79.598774802 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : configmap "kube-rbac-proxy" not found
Jul 16 21:51:11.794432 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:11.794376    2585 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:11.794432 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:11.794415    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls podName:bf883ff1-b068-427a-b5e7-d69cbf631112 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:19.794403476 +0000 UTC m=+79.598826797 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls") pod "insights-runtime-extractor-mfqnh" (UID: "bf883ff1-b068-427a-b5e7-d69cbf631112") : secret "insights-runtime-extractor-tls" not found
Jul 16 21:51:12.041996 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:12.041952    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:12.186474 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:12.186366    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-5wx6j" event={"ID":"00df8ce7-3aa0-4336-a467-766ba77730b8","Type":"ContainerStarted","Data":"6c737df1fe1a3e58abb0b6144b940063b009187477b60927055a1468a24480c5"}
Jul 16 21:51:12.214290 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:12.214235    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-5wx6j" podStartSLOduration=66.97709324 podStartE2EDuration="1m11.214214519s" totalImagesPullingTime="4.237121279s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:12.213575662 +0000 UTC m=+72.017998990" watchObservedRunningTime="2026-07-16 21:51:12.214214519 +0000 UTC m=+72.018637849"
Jul 16 21:51:12.268012 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:12.267975    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:12.466875 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:12.466780    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:13.487317 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:13.487283    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:13.561548 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:13.561511    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:13.645657 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:13.645620    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:13.915555 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:13.915445    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:13.918071 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:13.918047    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"image-registry-6c9998cdd8-f8hkr\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") " pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:14.016201 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.016157    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:51:14.018744 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.018718    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/1defe86c-f9b6-4b89-b263-3f8d84c15935-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-76cf8966f8-grzb8\" (UID: \"1defe86c-f9b6-4b89-b263-3f8d84c15935\") " pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:51:14.044448 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.044405    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.044616 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.044547    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-nmm9q\""
Jul 16 21:51:14.052661 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.052638    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:14.150190 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.150156    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-zkdw2\""
Jul 16 21:51:14.157585 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.157545    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"
Jul 16 21:51:14.251072 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.250764    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-6c9998cdd8-f8hkr"]
Jul 16 21:51:14.253185 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:14.253149    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd22bc564_2ae4_44fa_ad34_4a3761415500.slice/crio-711a72f5a69c6ace1738e6f987da0869da2e6db26aa6b6cd964b8c750780f529 WatchSource:0}: Error finding container 711a72f5a69c6ace1738e6f987da0869da2e6db26aa6b6cd964b8c750780f529: Status 404 returned error can't find the container with id 711a72f5a69c6ace1738e6f987da0869da2e6db26aa6b6cd964b8c750780f529
Jul 16 21:51:14.311592 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.311567    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8"]
Jul 16 21:51:14.315452 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:14.315426    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1defe86c_f9b6_4b89_b263_3f8d84c15935.slice/crio-cd6f866f314088db5a07ccc4a98eb23e341bb797a6c94d128fa73d5551d69144 WatchSource:0}: Error finding container cd6f866f314088db5a07ccc4a98eb23e341bb797a6c94d128fa73d5551d69144: Status 404 returned error can't find the container with id cd6f866f314088db5a07ccc4a98eb23e341bb797a6c94d128fa73d5551d69144
Jul 16 21:51:14.322935 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.322905    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.467928 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.467814    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.675593 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.675556    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:14.773415 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:14.773365    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:15.197995 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.197902    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8" event={"ID":"1defe86c-f9b6-4b89-b263-3f8d84c15935","Type":"ContainerStarted","Data":"cd6f866f314088db5a07ccc4a98eb23e341bb797a6c94d128fa73d5551d69144"}
Jul 16 21:51:15.199789 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.199756    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr" event={"ID":"d22bc564-2ae4-44fa-ad34-4a3761415500","Type":"ContainerStarted","Data":"4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45"}
Jul 16 21:51:15.199941 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.199811    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr" event={"ID":"d22bc564-2ae4-44fa-ad34-4a3761415500","Type":"ContainerStarted","Data":"711a72f5a69c6ace1738e6f987da0869da2e6db26aa6b6cd964b8c750780f529"}
Jul 16 21:51:15.199941 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.199850    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:15.224112 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.223992    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr" podStartSLOduration=17.223972192 podStartE2EDuration="17.223972192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:15.222886277 +0000 UTC m=+75.027309605" watchObservedRunningTime="2026-07-16 21:51:15.223972192 +0000 UTC m=+75.028395518"
Jul 16 21:51:15.243452 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.243415    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:15.486912 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.486873    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:15.583899 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.583856    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:15.703030 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:15.702975    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:16.096596 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:16.096561    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:16.205308 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:16.205275    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8" event={"ID":"1defe86c-f9b6-4b89-b263-3f8d84c15935","Type":"ContainerStarted","Data":"98aa2998583820e2326a859f999ae04a1e69f1dece9c2c55b12547a635f439a9"}
Jul 16 21:51:16.239560 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:16.239504    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-76cf8966f8-grzb8" podStartSLOduration=16.499834976 podStartE2EDuration="18.239488868s" totalImagesPullingTime="1.739653892s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:16.239213847 +0000 UTC m=+76.043637173" watchObservedRunningTime="2026-07-16 21:51:16.239488868 +0000 UTC m=+76.043912194"
Jul 16 21:51:16.252891 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:16.252854    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:16.437018 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:16.436922    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:16.624243 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:16.624207    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:16.824631 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:16.824591    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:17.300452 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:17.300412    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:17.651289 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:17.651194    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:17.841814 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:17.841779    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:18.048412 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:18.048370    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:18.659828 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:18.659776    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:18.662287 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:18.662257    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/c9469eba-c73f-4135-aa80-f12cf16ee555-networking-console-plugin-cert\") pod \"networking-console-plugin-7bc7489cb5-l262c\" (UID: \"c9469eba-c73f-4135-aa80-f12cf16ee555\") " pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:18.718736 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:18.718695    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"
Jul 16 21:51:18.849310 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:18.849269    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-7bc7489cb5-l262c"]
Jul 16 21:51:18.852346 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:18.852312    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9469eba_c73f_4135_aa80_f12cf16ee555.slice/crio-634a46be3aec93324c88ddb135a0c6925a814279f7c16e883e1dc4b8fe7a233a WatchSource:0}: Error finding container 634a46be3aec93324c88ddb135a0c6925a814279f7c16e883e1dc4b8fe7a233a: Status 404 returned error can't find the container with id 634a46be3aec93324c88ddb135a0c6925a814279f7c16e883e1dc4b8fe7a233a
Jul 16 21:51:19.215344 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:19.215309    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c" event={"ID":"c9469eba-c73f-4135-aa80-f12cf16ee555","Type":"ContainerStarted","Data":"634a46be3aec93324c88ddb135a0c6925a814279f7c16e883e1dc4b8fe7a233a"}
Jul 16 21:51:19.868647 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:19.868608    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:19.869274 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:19.868664    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:19.869383 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:19.869351    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/bf883ff1-b068-427a-b5e7-d69cbf631112-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:19.871608 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:19.871577    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/bf883ff1-b068-427a-b5e7-d69cbf631112-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-mfqnh\" (UID: \"bf883ff1-b068-427a-b5e7-d69cbf631112\") " pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:19.985884 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:19.985852    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-mfqnh"
Jul 16 21:51:20.245946 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:20.245904    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-mfqnh"]
Jul 16 21:51:20.249551 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:20.249521    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf883ff1_b068_427a_b5e7_d69cbf631112.slice/crio-242f7966edb766fc2f83eaf096c7079b94f959290f8e608ecf444628e13c879d WatchSource:0}: Error finding container 242f7966edb766fc2f83eaf096c7079b94f959290f8e608ecf444628e13c879d: Status 404 returned error can't find the container with id 242f7966edb766fc2f83eaf096c7079b94f959290f8e608ecf444628e13c879d
Jul 16 21:51:21.225426 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:21.225385    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c" event={"ID":"c9469eba-c73f-4135-aa80-f12cf16ee555","Type":"ContainerStarted","Data":"4553d16ec640f65f1cc5cfebb7c81bfeb057bb4875174478aa824151483c5e34"}
Jul 16 21:51:21.226918 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:21.226887    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mfqnh" event={"ID":"bf883ff1-b068-427a-b5e7-d69cbf631112","Type":"ContainerStarted","Data":"242f7966edb766fc2f83eaf096c7079b94f959290f8e608ecf444628e13c879d"}
Jul 16 21:51:21.247703 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:21.247631    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-7bc7489cb5-l262c" podStartSLOduration=17.949967511 podStartE2EDuration="19.247611729s" totalImagesPullingTime="1.297644218s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:21.245669078 +0000 UTC m=+81.050092403" watchObservedRunningTime="2026-07-16 21:51:21.247611729 +0000 UTC m=+81.052035055"
Jul 16 21:51:22.821648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:22.821510    2585 scope.go:117] "RemoveContainer" containerID="bdf7c9d421b8e3b22c62f3e67acdea4fd6a9d51240476c507ae4e6125e8bab28"
Jul 16 21:51:22.835870 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:22.835813    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:25.247622 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:25.247592    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 21:51:25.248034 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:25.247719    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" event={"ID":"aa88b577-be62-48bb-bf4d-2eadb3b2d857","Type":"ContainerStarted","Data":"469576fc8d0fad0f807fd90ad7df9330438ce4aa5ba9d05cb9eac256d2aaae61"}
Jul 16 21:51:25.248279 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:25.248257    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:51:25.249746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:25.249715    2585 patch_prober.go:28] interesting pod/console-operator-565b7b569d-5jwjh container/console-operator namespace/openshift-console-operator: Readiness probe status=failure output="Get \"https://10.134.0.13:8443/readyz\": dial tcp 10.134.0.13:8443: connect: connection refused" start-of-body=
Jul 16 21:51:25.249845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:25.249772    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" podUID="aa88b577-be62-48bb-bf4d-2eadb3b2d857" containerName="console-operator" probeResult="failure" output="Get \"https://10.134.0.13:8443/readyz\": dial tcp 10.134.0.13:8443: connect: connection refused"
Jul 16 21:51:25.280354 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:25.280297    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh" podStartSLOduration=23.352732062 podStartE2EDuration="27.280278111s" totalImagesPullingTime="3.927546049s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:25.279630484 +0000 UTC m=+85.084053810" watchObservedRunningTime="2026-07-16 21:51:25.280278111 +0000 UTC m=+85.084701437"
Jul 16 21:51:26.255884 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:26.255840    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mfqnh" event={"ID":"bf883ff1-b068-427a-b5e7-d69cbf631112","Type":"ContainerStarted","Data":"ce7ddc73db3287af23cfd4424fb6922c00d374c2f23309f64a57d72726784ba0"}
Jul 16 21:51:26.260875 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:26.260843    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-565b7b569d-5jwjh"
Jul 16 21:51:26.287231 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:26.287180    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:26.429403 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:26.429358    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:26.506391 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:26.506313    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:26.684291 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:26.684251    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:26.802008 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:26.801921    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:27.038495 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:27.038456    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:27.125580 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:27.125494    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:27.230354 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:27.230301    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:27.305403 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:27.305358    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:27.641138 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:27.641071    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:27.850732 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:27.850645    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:27.989869 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:27.989834    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:29.063483 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:29.063437    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:29.269321 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:29.269284    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mfqnh" event={"ID":"bf883ff1-b068-427a-b5e7-d69cbf631112","Type":"ContainerStarted","Data":"5352c0b31c9a42117bdb2721d64d1139c701bbb733ccb516c7ee1f55463a13f6"}
Jul 16 21:51:30.573920 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.573882    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c9998cdd8-f8hkr"]
Jul 16 21:51:30.579864 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.579803    2585 patch_prober.go:28] interesting pod/image-registry-6c9998cdd8-f8hkr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 21:51:30.580053 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.579890    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr" podUID="d22bc564-2ae4-44fa-ad34-4a3761415500" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 21:51:30.597706 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.597662    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"]
Jul 16 21:51:30.602651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.602620    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-5bc67dcd46-dmksn"]
Jul 16 21:51:30.602816 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.602782    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"
Jul 16 21:51:30.605412 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.605384    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-td27h\""
Jul 16 21:51:30.605709 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.605687    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-dmksn"
Jul 16 21:51:30.605845 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.605767    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 16 21:51:30.608124 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.608088    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-z7xbc\""
Jul 16 21:51:30.608323 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.608307    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 16 21:51:30.608489 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.608466    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 16 21:51:30.614961 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.614934    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"]
Jul 16 21:51:30.616370 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.616339    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-dmksn"]
Jul 16 21:51:30.754184 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.754147    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:30.772066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.772028    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s7hvb\" (UniqueName: \"kubernetes.io/projected/c4a90ba1-3a1e-42b3-91a4-287b04054e2d-kube-api-access-s7hvb\") pod \"downloads-5bc67dcd46-dmksn\" (UID: \"c4a90ba1-3a1e-42b3-91a4-287b04054e2d\") " pod="openshift-console/downloads-5bc67dcd46-dmksn"
Jul 16 21:51:30.772462 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.772428    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f7e248a8-1d54-411c-8966-a727b77419f1-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6d7cf\" (UID: \"f7e248a8-1d54-411c-8966-a727b77419f1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"
Jul 16 21:51:30.854951 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.854839    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:30.873594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.873560    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f7e248a8-1d54-411c-8966-a727b77419f1-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6d7cf\" (UID: \"f7e248a8-1d54-411c-8966-a727b77419f1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"
Jul 16 21:51:30.873783 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.873629    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s7hvb\" (UniqueName: \"kubernetes.io/projected/c4a90ba1-3a1e-42b3-91a4-287b04054e2d-kube-api-access-s7hvb\") pod \"downloads-5bc67dcd46-dmksn\" (UID: \"c4a90ba1-3a1e-42b3-91a4-287b04054e2d\") " pod="openshift-console/downloads-5bc67dcd46-dmksn"
Jul 16 21:51:30.876617 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.876586    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/f7e248a8-1d54-411c-8966-a727b77419f1-tls-certificates\") pod \"prometheus-operator-admission-webhook-85574b5f49-6d7cf\" (UID: \"f7e248a8-1d54-411c-8966-a727b77419f1\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"
Jul 16 21:51:30.884812 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.884778    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s7hvb\" (UniqueName: \"kubernetes.io/projected/c4a90ba1-3a1e-42b3-91a4-287b04054e2d-kube-api-access-s7hvb\") pod \"downloads-5bc67dcd46-dmksn\" (UID: \"c4a90ba1-3a1e-42b3-91a4-287b04054e2d\") " pod="openshift-console/downloads-5bc67dcd46-dmksn"
Jul 16 21:51:30.915513 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.915466    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"
Jul 16 21:51:30.922868 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.922843    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-5bc67dcd46-dmksn"
Jul 16 21:51:30.955598 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:30.955550    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:31.482923 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.482887    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-784dfc5755-dtzzx"]
Jul 16 21:51:31.498544 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.498511    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.501551 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.501435    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 16 21:51:31.501551 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.501461    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 16 21:51:31.501761 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.501610    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 16 21:51:31.501761 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.501608    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 16 21:51:31.501761 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.501733    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-khf9x\""
Jul 16 21:51:31.502627 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.502607    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 16 21:51:31.508171 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.507762    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784dfc5755-dtzzx"]
Jul 16 21:51:31.680533 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.680497    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-oauth-config\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.680982 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.680555    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-config\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.680982 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.680631    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-oauth-serving-cert\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.680982 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.680674    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwwn6\" (UniqueName: \"kubernetes.io/projected/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-kube-api-access-pwwn6\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.680982 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.680751    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-serving-cert\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.680982 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.680788    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-service-ca\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.781482 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.781392    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-oauth-serving-cert\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.781482 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.781441    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwwn6\" (UniqueName: \"kubernetes.io/projected/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-kube-api-access-pwwn6\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.781738 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.781492    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-serving-cert\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.781738 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.781512    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-service-ca\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.781738 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.781572    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-oauth-config\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.781738 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.781618    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-config\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.782315 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.782287    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-service-ca\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.782315 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.782305    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-oauth-serving-cert\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.782503 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.782347    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-config\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.784994 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.784961    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-serving-cert\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.785149 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.785056    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-oauth-config\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.790791 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.790759    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwwn6\" (UniqueName: \"kubernetes.io/projected/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-kube-api-access-pwwn6\") pod \"console-784dfc5755-dtzzx\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") " pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:31.811712 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:31.811648    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:32.376912 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:32.376873    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"]
Jul 16 21:51:32.380040 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:32.380008    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf7e248a8_1d54_411c_8966_a727b77419f1.slice/crio-f68753db4b65faf879aff7a7c187bf8365b78d03d35c0a8b823bec4d6b8bdfea WatchSource:0}: Error finding container f68753db4b65faf879aff7a7c187bf8365b78d03d35c0a8b823bec4d6b8bdfea: Status 404 returned error can't find the container with id f68753db4b65faf879aff7a7c187bf8365b78d03d35c0a8b823bec4d6b8bdfea
Jul 16 21:51:32.602498 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:32.602456    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-784dfc5755-dtzzx"]
Jul 16 21:51:32.606057 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:32.606029    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-5bc67dcd46-dmksn"]
Jul 16 21:51:32.606599 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:32.606567    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1eb9826_cdfe_40ac_83bb_a7c40c5a3cf2.slice/crio-c85d0ad0cf3cb75ddd415200cd3c1fba139adfd94dea8718a0bd760033f43c4f WatchSource:0}: Error finding container c85d0ad0cf3cb75ddd415200cd3c1fba139adfd94dea8718a0bd760033f43c4f: Status 404 returned error can't find the container with id c85d0ad0cf3cb75ddd415200cd3c1fba139adfd94dea8718a0bd760033f43c4f
Jul 16 21:51:32.608732 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:32.608670    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4a90ba1_3a1e_42b3_91a4_287b04054e2d.slice/crio-291e3602161024fbe667bef0eab924187a38c4e5bb640abe4b5fe1be9fc6ca53 WatchSource:0}: Error finding container 291e3602161024fbe667bef0eab924187a38c4e5bb640abe4b5fe1be9fc6ca53: Status 404 returned error can't find the container with id 291e3602161024fbe667bef0eab924187a38c4e5bb640abe4b5fe1be9fc6ca53
Jul 16 21:51:33.289967 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:33.289890    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf" event={"ID":"f7e248a8-1d54-411c-8966-a727b77419f1","Type":"ContainerStarted","Data":"f68753db4b65faf879aff7a7c187bf8365b78d03d35c0a8b823bec4d6b8bdfea"}
Jul 16 21:51:33.291821 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:33.291785    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784dfc5755-dtzzx" event={"ID":"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2","Type":"ContainerStarted","Data":"c85d0ad0cf3cb75ddd415200cd3c1fba139adfd94dea8718a0bd760033f43c4f"}
Jul 16 21:51:33.293304 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:33.293220    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-dmksn" event={"ID":"c4a90ba1-3a1e-42b3-91a4-287b04054e2d","Type":"ContainerStarted","Data":"291e3602161024fbe667bef0eab924187a38c4e5bb640abe4b5fe1be9fc6ca53"}
Jul 16 21:51:33.295886 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:33.295831    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-mfqnh" event={"ID":"bf883ff1-b068-427a-b5e7-d69cbf631112","Type":"ContainerStarted","Data":"0fc6554cc43cf00377af51880b8b76765ba2e181f2134e02cb806610cf518a0d"}
Jul 16 21:51:33.318223 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:33.318152    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-mfqnh" podStartSLOduration=17.399708345 podStartE2EDuration="29.318134112s" totalImagesPullingTime="11.918425767s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:33.316642935 +0000 UTC m=+93.121066262" watchObservedRunningTime="2026-07-16 21:51:33.318134112 +0000 UTC m=+93.122557438"
Jul 16 21:51:34.303495 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.303283    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf" event={"ID":"f7e248a8-1d54-411c-8966-a727b77419f1","Type":"ContainerStarted","Data":"8d54db087b4f4f87ce58f10e8b7bb7cfc7550103091591de40c8a5b740627c85"}
Jul 16 21:51:34.303495 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.303372    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"
Jul 16 21:51:34.309713 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.309676    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf"
Jul 16 21:51:34.323996 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.323943    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-85574b5f49-6d7cf" podStartSLOduration=2.665676517 podStartE2EDuration="4.323920038s" totalImagesPullingTime="1.658243521s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:34.321936139 +0000 UTC m=+94.126359470" watchObservedRunningTime="2026-07-16 21:51:34.323920038 +0000 UTC m=+94.128343367"
Jul 16 21:51:34.765020 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.764968    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:34.870918 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.870870    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"]
Jul 16 21:51:34.890175 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.890132    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"]
Jul 16 21:51:34.890357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.890246    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:34.896044 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.896017    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 16 21:51:34.896213 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.896052    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 16 21:51:34.897330 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.897305    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-plfx7\""
Jul 16 21:51:34.897452 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.897335    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 16 21:51:34.911524 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.910986    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sh4h\" (UniqueName: \"kubernetes.io/projected/dead8117-40cc-4617-b2e7-560ec9ad992e-kube-api-access-2sh4h\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:34.911524 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.911050    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dead8117-40cc-4617-b2e7-560ec9ad992e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:34.911524 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.911230    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dead8117-40cc-4617-b2e7-560ec9ad992e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:34.911524 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.911310    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dead8117-40cc-4617-b2e7-560ec9ad992e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:34.911524 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:34.911431    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:35.012700 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.012657    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sh4h\" (UniqueName: \"kubernetes.io/projected/dead8117-40cc-4617-b2e7-560ec9ad992e-kube-api-access-2sh4h\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:35.012908 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.012718    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dead8117-40cc-4617-b2e7-560ec9ad992e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:35.012908 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.012882    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dead8117-40cc-4617-b2e7-560ec9ad992e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:35.013020 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.012927    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dead8117-40cc-4617-b2e7-560ec9ad992e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:35.013632 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.013602    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dead8117-40cc-4617-b2e7-560ec9ad992e-metrics-client-ca\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:35.016229 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.016172    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/dead8117-40cc-4617-b2e7-560ec9ad992e-prometheus-operator-tls\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:35.016229 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.016172    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dead8117-40cc-4617-b2e7-560ec9ad992e-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:35.023980 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.023934    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sh4h\" (UniqueName: \"kubernetes.io/projected/dead8117-40cc-4617-b2e7-560ec9ad992e-kube-api-access-2sh4h\") pod \"prometheus-operator-5945b5cfd-7pvwx\" (UID: \"dead8117-40cc-4617-b2e7-560ec9ad992e\") " pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:35.032617 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.032518    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:35.203614 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:35.203580    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"
Jul 16 21:51:36.058275 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:36.058237    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx"]
Jul 16 21:51:36.062319 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:36.062281    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddead8117_40cc_4617_b2e7_560ec9ad992e.slice/crio-47395f65776681120b2753d3bf345671c947bac9f1574f16ff312bda9f4de35e WatchSource:0}: Error finding container 47395f65776681120b2753d3bf345671c947bac9f1574f16ff312bda9f4de35e: Status 404 returned error can't find the container with id 47395f65776681120b2753d3bf345671c947bac9f1574f16ff312bda9f4de35e
Jul 16 21:51:36.264937 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:36.264893    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:36.315558 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:36.315521    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx" event={"ID":"dead8117-40cc-4617-b2e7-560ec9ad992e","Type":"ContainerStarted","Data":"47395f65776681120b2753d3bf345671c947bac9f1574f16ff312bda9f4de35e"}
Jul 16 21:51:36.317495 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:36.317463    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784dfc5755-dtzzx" event={"ID":"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2","Type":"ContainerStarted","Data":"c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c"}
Jul 16 21:51:36.344937 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:36.344875    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-784dfc5755-dtzzx" podStartSLOduration=1.99957012 podStartE2EDuration="5.344856288s" totalImagesPullingTime="3.345286168s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:36.343556887 +0000 UTC m=+96.147980213" watchObservedRunningTime="2026-07-16 21:51:36.344856288 +0000 UTC m=+96.149279592"
Jul 16 21:51:36.366115 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:36.366069    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:37.325032 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:37.324991    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx" event={"ID":"dead8117-40cc-4617-b2e7-560ec9ad992e","Type":"ContainerStarted","Data":"c2d4bedcddc95a10c9f9318a827e7108a3199ae5f445c4ea0a4cfe8d75a0bfbf"}
Jul 16 21:51:37.634638 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:37.634595    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:38.175110 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:38.175076    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-dhp2x"
Jul 16 21:51:38.330809 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:38.330764    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx" event={"ID":"dead8117-40cc-4617-b2e7-560ec9ad992e","Type":"ContainerStarted","Data":"46758be71941cbf67a7707229e6b9ac9463a76cb92acce93fc8b1551d8d23777"}
Jul 16 21:51:38.360403 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:38.360340    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-5945b5cfd-7pvwx" podStartSLOduration=3.1979539 podStartE2EDuration="4.360323627s" totalImagesPullingTime="1.162369727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:38.359340072 +0000 UTC m=+98.163763399" watchObservedRunningTime="2026-07-16 21:51:38.360323627 +0000 UTC m=+98.164746952"
Jul 16 21:51:39.149504 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:39.149464    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:39.238874 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:39.238838    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:40.015843 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.015755    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:40.185766 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.185724    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:40.413314 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.413201    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:40.533453 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.533415    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-plpcr"]
Jul 16 21:51:40.537465 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.537434    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.549987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.549957    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\""
Jul 16 21:51:40.558007 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.557977    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-tjmgl\""
Jul 16 21:51:40.558216 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.557977    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\""
Jul 16 21:51:40.558216 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.557977    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\""
Jul 16 21:51:40.565965 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.565933    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.566150 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.565998    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.566150 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.566117    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gdkzg\" (UniqueName: \"kubernetes.io/projected/c6fc99cf-c316-4484-a451-c51854b1727e-kube-api-access-gdkzg\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.566275 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.566152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6fc99cf-c316-4484-a451-c51854b1727e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.566275 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.566200    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6fc99cf-c316-4484-a451-c51854b1727e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.566275 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.566227    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.579355 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.579312    2585 patch_prober.go:28] interesting pod/image-registry-6c9998cdd8-f8hkr container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 16 21:51:40.579574 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.579383    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr" podUID="d22bc564-2ae4-44fa-ad34-4a3761415500" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 16 21:51:40.646905 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.646871    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-jvqs7"]
Jul 16 21:51:40.648306 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.648270    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:40.650518 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.650497    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.665382 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.665302    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 16 21:51:40.668559 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.668456    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-plpcr"]
Jul 16 21:51:40.668753 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.668721    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6fc99cf-c316-4484-a451-c51854b1727e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.668830 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.668781    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.668884 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.668869    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.668945 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.668930    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.670712 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.669022    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gdkzg\" (UniqueName: \"kubernetes.io/projected/c6fc99cf-c316-4484-a451-c51854b1727e-kube-api-access-gdkzg\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.670712 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.669067    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6fc99cf-c316-4484-a451-c51854b1727e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.670712 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.669888    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/c6fc99cf-c316-4484-a451-c51854b1727e-metrics-client-ca\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.670712 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.670189    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/c6fc99cf-c316-4484-a451-c51854b1727e-volume-directive-shadow\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.674127 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.671083    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.674127 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.671303    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 16 21:51:40.674127 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.671314    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-drdzb\""
Jul 16 21:51:40.674127 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.671631    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 16 21:51:40.674127 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.673505    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.676452 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.676429    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/c6fc99cf-c316-4484-a451-c51854b1727e-kube-state-metrics-tls\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.710164 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.710122    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gdkzg\" (UniqueName: \"kubernetes.io/projected/c6fc99cf-c316-4484-a451-c51854b1727e-kube-api-access-gdkzg\") pod \"kube-state-metrics-56c5594674-plpcr\" (UID: \"c6fc99cf-c316-4484-a451-c51854b1727e\") " pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.769525 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769487    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-tls\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.769753 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769553    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-metrics-client-ca\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.769753 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769575    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7l7vm\" (UniqueName: \"kubernetes.io/projected/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-kube-api-access-7l7vm\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.769753 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769632    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.769753 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769722    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-textfile\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.769993 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769760    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.769993 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769816    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-sys\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.769993 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769838    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-root\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.769993 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.769881    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-wtmp\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.822368 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.822322    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:40.848446 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.848409    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr"
Jul 16 21:51:40.871148 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871088    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-wtmp\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871328 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871177    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-tls\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871328 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-metrics-client-ca\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871328 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871255    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7l7vm\" (UniqueName: \"kubernetes.io/projected/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-kube-api-access-7l7vm\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871328 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871285    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-wtmp\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871541 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871351    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871541 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871395    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-textfile\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871541 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871420    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871541 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871471    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-sys\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871541 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871496    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-root\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.871905 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.871578    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-root\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.873385 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.872515    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-sys\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.873385 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.872700    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-metrics-client-ca\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.873385 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.872843    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-textfile\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.873385 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.873353    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-accelerators-collector-config\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.874541 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.874516    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-tls\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.875627 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.875604    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.898633 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.898559    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7l7vm\" (UniqueName: \"kubernetes.io/projected/34db72e0-c3d3-443e-9b0b-d89ea20e4afb-kube-api-access-7l7vm\") pod \"node-exporter-jvqs7\" (UID: \"34db72e0-c3d3-443e-9b0b-d89ea20e4afb\") " pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.952398 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.952363    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:40.961346 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:40.961314    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-jvqs7"
Jul 16 21:51:40.981277 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:40.981086    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34db72e0_c3d3_443e_9b0b_d89ea20e4afb.slice/crio-a061ef65d4f11576eccf7a287f64ced9f0848374de7da6679602a51cd4d3c182 WatchSource:0}: Error finding container a061ef65d4f11576eccf7a287f64ced9f0848374de7da6679602a51cd4d3c182: Status 404 returned error can't find the container with id a061ef65d4f11576eccf7a287f64ced9f0848374de7da6679602a51cd4d3c182
Jul 16 21:51:41.046693 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:41.046642    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6fc99cf_c316_4484_a451_c51854b1727e.slice/crio-7d14d37265b42183f31692bdff84b22a9130931171f2eb3aa369e74f058174fa WatchSource:0}: Error finding container 7d14d37265b42183f31692bdff84b22a9130931171f2eb3aa369e74f058174fa: Status 404 returned error can't find the container with id 7d14d37265b42183f31692bdff84b22a9130931171f2eb3aa369e74f058174fa
Jul 16 21:51:41.047144 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.046691    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-56c5594674-plpcr"]
Jul 16 21:51:41.190228 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.190168    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:41.345668 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.345624    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr" event={"ID":"c6fc99cf-c316-4484-a451-c51854b1727e","Type":"ContainerStarted","Data":"7d14d37265b42183f31692bdff84b22a9130931171f2eb3aa369e74f058174fa"}
Jul 16 21:51:41.347073 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.347036    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqs7" event={"ID":"34db72e0-c3d3-443e-9b0b-d89ea20e4afb","Type":"ContainerStarted","Data":"a061ef65d4f11576eccf7a287f64ced9f0848374de7da6679602a51cd4d3c182"}
Jul 16 21:51:41.412431 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.412190    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:41.475865 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.475790    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:51:41.479247 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.479183    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/90027345-b94e-49c5-8d72-0bff02dcdf87-metrics-tls\") pod \"dns-default-rpn85\" (UID: \"90027345-b94e-49c5-8d72-0bff02dcdf87\") " pod="openshift-dns/dns-default-rpn85"
Jul 16 21:51:41.531491 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.531451    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-6n6rt\""
Jul 16 21:51:41.540584 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.540506    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-rpn85"
Jul 16 21:51:41.610581 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.610443    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:41.753977 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:41.753939    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90027345_b94e_49c5_8d72_0bff02dcdf87.slice/crio-a025722de17ab99e7b3e3be0c763fec9872e51a2799df8050f4fd8bf76d32329 WatchSource:0}: Error finding container a025722de17ab99e7b3e3be0c763fec9872e51a2799df8050f4fd8bf76d32329: Status 404 returned error can't find the container with id a025722de17ab99e7b3e3be0c763fec9872e51a2799df8050f4fd8bf76d32329
Jul 16 21:51:41.756548 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.756512    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-rpn85"]
Jul 16 21:51:41.761750 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.761674    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:41.812810 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.812550    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:41.814151 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.814041    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:51:41.816261 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.816222    2585 patch_prober.go:28] interesting pod/console-784dfc5755-dtzzx container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused" start-of-body=
Jul 16 21:51:41.816430 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:41.816292    2585 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-784dfc5755-dtzzx" podUID="a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" containerName="console" probeResult="failure" output="Get \"https://10.134.0.18:8443/health\": dial tcp 10.134.0.18:8443: connect: connection refused"
Jul 16 21:51:42.082900 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.081958    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-584fdbb98-zhg6n"]
Jul 16 21:51:42.116974 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.116898    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584fdbb98-zhg6n"]
Jul 16 21:51:42.117163 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.116992    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.155990 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.155945    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 16 21:51:42.184963 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.184924    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-serving-cert\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.185165 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.184994    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-service-ca\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.185165 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.185051    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-trusted-ca-bundle\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.185165 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.185113    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvkmk\" (UniqueName: \"kubernetes.io/projected/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-kube-api-access-hvkmk\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.185165 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.185153    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-oauth-config\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.185348 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.185169    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-oauth-serving-cert\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.185348 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.185228    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-config\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.204929 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.204895    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:42.285722 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.285681    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-service-ca\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.285722 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.285738    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-trusted-ca-bundle\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.285984 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.285784    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvkmk\" (UniqueName: \"kubernetes.io/projected/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-kube-api-access-hvkmk\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.285984 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.285823    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-oauth-config\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.285984 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.285843    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-oauth-serving-cert\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.285984 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.285889    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-config\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.285984 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.285935    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-serving-cert\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.286633 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.286585    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-service-ca\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.286847 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.286823    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-oauth-serving-cert\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.287134 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.287054    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-config\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.287134 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.287114    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-trusted-ca-bundle\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.289016 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.288850    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-serving-cert\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.289016 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.288908    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-oauth-config\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.306643 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.306606    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvkmk\" (UniqueName: \"kubernetes.io/projected/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-kube-api-access-hvkmk\") pod \"console-584fdbb98-zhg6n\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") " pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.353392 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.353297    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rpn85" event={"ID":"90027345-b94e-49c5-8d72-0bff02dcdf87","Type":"ContainerStarted","Data":"a025722de17ab99e7b3e3be0c763fec9872e51a2799df8050f4fd8bf76d32329"}
Jul 16 21:51:42.433773 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.433733    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:51:42.726931 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.726782    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:42.844509 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:42.844470    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:43.486745 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.486709    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:43.553826 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.553784    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-86656b897-m6jt9"]
Jul 16 21:51:43.558542 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.558502    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.571782 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.571758    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Jul 16 21:51:43.572643 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.572613    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Jul 16 21:51:43.573393 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.573372    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Jul 16 21:51:43.573489 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.573438    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Jul 16 21:51:43.573489 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.573470    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-zhrbv\""
Jul 16 21:51:43.580628 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.580600    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Jul 16 21:51:43.582741 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.582715    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-oi70v3j539gv\""
Jul 16 21:51:43.592162 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.592132    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86656b897-m6jt9"]
Jul 16 21:51:43.600005 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.599967    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-tls\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.600201 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.600030    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-grpc-tls\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.600201 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.600115    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.600201 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.600176    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e0a640-0edc-4cb6-add3-0e5a25953d8a-metrics-client-ca\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.600357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.600229    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.600357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.600295    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.600442 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.600364    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.600490 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.600442    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm2d7\" (UniqueName: \"kubernetes.io/projected/69e0a640-0edc-4cb6-add3-0e5a25953d8a-kube-api-access-wm2d7\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.652648 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.652590    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:43.701617 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.701572    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.701813 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.701641    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.701813 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.701671    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.701813 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.701713    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm2d7\" (UniqueName: \"kubernetes.io/projected/69e0a640-0edc-4cb6-add3-0e5a25953d8a-kube-api-access-wm2d7\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.701813 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.701746    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-tls\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.701813 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.701784    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-grpc-tls\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.701813 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.701813    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.702170 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.702014    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e0a640-0edc-4cb6-add3-0e5a25953d8a-metrics-client-ca\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.703024 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.702966    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/69e0a640-0edc-4cb6-add3-0e5a25953d8a-metrics-client-ca\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.704896 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.704864    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.705113 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.705068    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-grpc-tls\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.705822 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.705583    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.705822 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.705775    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-tls\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.706014 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.705986    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.706014 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.705994    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/69e0a640-0edc-4cb6-add3-0e5a25953d8a-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.724558 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.724528    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm2d7\" (UniqueName: \"kubernetes.io/projected/69e0a640-0edc-4cb6-add3-0e5a25953d8a-kube-api-access-wm2d7\") pod \"thanos-querier-86656b897-m6jt9\" (UID: \"69e0a640-0edc-4cb6-add3-0e5a25953d8a\") " pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.736995 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.736898    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:43.872213 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.872176    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:43.948083 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:43.948039    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:45.116730 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.116693    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:45.129993 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.129957    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"]
Jul 16 21:51:45.135509 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.133997    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:45.142249 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.142222    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-cert\""
Jul 16 21:51:45.142415 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.142222    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"monitoring-plugin-dockercfg-twv9t\""
Jul 16 21:51:45.150060 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.150033    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"]
Jul 16 21:51:45.199595 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.199560    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"]
Jul 16 21:51:45.203061 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.203034    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.212623 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.212591    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Jul 16 21:51:45.213880 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.213853    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-nmgp9\""
Jul 16 21:51:45.214220 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.214198    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 16 21:51:45.214519 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.214496    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Jul 16 21:51:45.214685 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.214668    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Jul 16 21:51:45.214786 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.214767    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-6fnbjgfujdhsf\""
Jul 16 21:51:45.216159 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.216132    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9476ce3d-86fb-4e4e-bede-d721052135d4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-56b6b\" (UID: \"9476ce3d-86fb-4e4e-bede-d721052135d4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:45.216263 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.216238    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hd2f\" (UniqueName: \"kubernetes.io/projected/9476ce3d-86fb-4e4e-bede-d721052135d4-kube-api-access-4hd2f\") pod \"monitoring-plugin-7b7b7bc8bf-56b6b\" (UID: \"9476ce3d-86fb-4e4e-bede-d721052135d4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:45.221168 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.221143    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"]
Jul 16 21:51:45.236881 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.236848    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:45.317716 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.317670    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9476ce3d-86fb-4e4e-bede-d721052135d4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-56b6b\" (UID: \"9476ce3d-86fb-4e4e-bede-d721052135d4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:45.317927 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.317732    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1d729d56-c88f-49ff-8882-e4e28d58859a-audit-log\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.317927 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.317762    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-client-ca-bundle\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.317927 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:45.317848    2585 secret.go:189] Couldn't get secret openshift-monitoring/monitoring-plugin-cert: secret "monitoring-plugin-cert" not found
Jul 16 21:51:45.317927 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.317857    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1d729d56-c88f-49ff-8882-e4e28d58859a-metrics-server-audit-profiles\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.318150 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:45.317941    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9476ce3d-86fb-4e4e-bede-d721052135d4-monitoring-plugin-cert podName:9476ce3d-86fb-4e4e-bede-d721052135d4 nodeName:}" failed. No retries permitted until 2026-07-16 21:51:45.817918371 +0000 UTC m=+105.622341674 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "monitoring-plugin-cert" (UniqueName: "kubernetes.io/secret/9476ce3d-86fb-4e4e-bede-d721052135d4-monitoring-plugin-cert") pod "monitoring-plugin-7b7b7bc8bf-56b6b" (UID: "9476ce3d-86fb-4e4e-bede-d721052135d4") : secret "monitoring-plugin-cert" not found
Jul 16 21:51:45.318150 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.318021    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hd2f\" (UniqueName: \"kubernetes.io/projected/9476ce3d-86fb-4e4e-bede-d721052135d4-kube-api-access-4hd2f\") pod \"monitoring-plugin-7b7b7bc8bf-56b6b\" (UID: \"9476ce3d-86fb-4e4e-bede-d721052135d4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:45.318150 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.318064    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-secret-metrics-server-client-certs\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.318150 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.318126    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-secret-metrics-server-tls\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.318341 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.318152    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cd9wx\" (UniqueName: \"kubernetes.io/projected/1d729d56-c88f-49ff-8882-e4e28d58859a-kube-api-access-cd9wx\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.318341 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.318234    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d729d56-c88f-49ff-8882-e4e28d58859a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.334497 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.334458    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hd2f\" (UniqueName: \"kubernetes.io/projected/9476ce3d-86fb-4e4e-bede-d721052135d4-kube-api-access-4hd2f\") pod \"monitoring-plugin-7b7b7bc8bf-56b6b\" (UID: \"9476ce3d-86fb-4e4e-bede-d721052135d4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:45.419020 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.418921    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-secret-metrics-server-tls\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.419020 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.418963    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cd9wx\" (UniqueName: \"kubernetes.io/projected/1d729d56-c88f-49ff-8882-e4e28d58859a-kube-api-access-cd9wx\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.419020 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.419017    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d729d56-c88f-49ff-8882-e4e28d58859a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.419484 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.419084    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1d729d56-c88f-49ff-8882-e4e28d58859a-audit-log\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.419484 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.419156    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-client-ca-bundle\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.419484 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.419191    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1d729d56-c88f-49ff-8882-e4e28d58859a-metrics-server-audit-profiles\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.419484 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.419410    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-secret-metrics-server-client-certs\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.420163 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.420138    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1d729d56-c88f-49ff-8882-e4e28d58859a-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.420415 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.420397    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/1d729d56-c88f-49ff-8882-e4e28d58859a-audit-log\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.421261 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.421237    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/1d729d56-c88f-49ff-8882-e4e28d58859a-metrics-server-audit-profiles\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.422085 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.422058    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-secret-metrics-server-tls\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.422219 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.422200    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-client-ca-bundle\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.422341 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.422317    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/1d729d56-c88f-49ff-8882-e4e28d58859a-secret-metrics-server-client-certs\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.436316 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.436282    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cd9wx\" (UniqueName: \"kubernetes.io/projected/1d729d56-c88f-49ff-8882-e4e28d58859a-kube-api-access-cd9wx\") pod \"metrics-server-69bf49fd5f-s7lk5\" (UID: \"1d729d56-c88f-49ff-8882-e4e28d58859a\") " pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.463719 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.463683    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:45.515839 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.515798    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:51:45.823950 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.823902    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9476ce3d-86fb-4e4e-bede-d721052135d4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-56b6b\" (UID: \"9476ce3d-86fb-4e4e-bede-d721052135d4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:45.826809 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:45.826779    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"monitoring-plugin-cert\" (UniqueName: \"kubernetes.io/secret/9476ce3d-86fb-4e4e-bede-d721052135d4-monitoring-plugin-cert\") pod \"monitoring-plugin-7b7b7bc8bf-56b6b\" (UID: \"9476ce3d-86fb-4e4e-bede-d721052135d4\") " pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:46.048917 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:46.048872    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:46.329964 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:46.329929    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:46.466413 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:46.466277    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:46.959986 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:46.959915    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:49.136050 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:49.136010    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:50.579044 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:50.579011    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:50.930020 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:50.929921    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-784dfc5755-dtzzx"]
Jul 16 21:51:50.997921 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:50.997880    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-5d47cb5bb8-q4f8k"]
Jul 16 21:51:51.001188 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.001149    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.020194 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.020154    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d47cb5bb8-q4f8k"]
Jul 16 21:51:51.081775 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.081733    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-console-config\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.081775 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.081778    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-trusted-ca-bundle\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.081987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.081811    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-oauth-config\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.081987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.081863    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vpcg\" (UniqueName: \"kubernetes.io/projected/222a7ebc-cdea-429c-9593-3ed72c4061af-kube-api-access-2vpcg\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.081987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.081910    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-oauth-serving-cert\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.081987 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.081932    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-serving-cert\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.082129 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.082014    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-service-ca\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.184154 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.182594    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vpcg\" (UniqueName: \"kubernetes.io/projected/222a7ebc-cdea-429c-9593-3ed72c4061af-kube-api-access-2vpcg\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.184154 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.182690    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-oauth-serving-cert\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.184154 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.182722    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-serving-cert\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.184154 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.182774    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-service-ca\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.184154 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.182839    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-console-config\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.184154 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.182864    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-trusted-ca-bundle\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.184154 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.182895    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-oauth-config\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.189378 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.189340    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-console-config\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.191853 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.191394    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-service-ca\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.191853 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.191804    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-trusted-ca-bundle\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.192058 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.191993    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-oauth-serving-cert\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.195413 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.195384    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-oauth-config\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.198419 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.198343    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-serving-cert\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.216921 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.216889    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vpcg\" (UniqueName: \"kubernetes.io/projected/222a7ebc-cdea-429c-9593-3ed72c4061af-kube-api-access-2vpcg\") pod \"console-5d47cb5bb8-q4f8k\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") " pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.319520 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.319199    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:51:51.329175 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.327804    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-86656b897-m6jt9"]
Jul 16 21:51:51.336408 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.336214    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-584fdbb98-zhg6n"]
Jul 16 21:51:51.338276 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:51.337821    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69e0a640_0edc_4cb6_add3_0e5a25953d8a.slice/crio-46e46a1479e0c3255689fdb5b034d2f5b3c281a90f2eb0f54044166946354909 WatchSource:0}: Error finding container 46e46a1479e0c3255689fdb5b034d2f5b3c281a90f2eb0f54044166946354909: Status 404 returned error can't find the container with id 46e46a1479e0c3255689fdb5b034d2f5b3c281a90f2eb0f54044166946354909
Jul 16 21:51:51.339687 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:51.339628    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod940c9e26_3574_4bd7_a47e_c8a8ea13ded9.slice/crio-e36dbd6c6d1e41e399e6973de4442f84a3371499314ce34b1d555881a12898de WatchSource:0}: Error finding container e36dbd6c6d1e41e399e6973de4442f84a3371499314ce34b1d555881a12898de: Status 404 returned error can't find the container with id e36dbd6c6d1e41e399e6973de4442f84a3371499314ce34b1d555881a12898de
Jul 16 21:51:51.365066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.365034    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"]
Jul 16 21:51:51.369877 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:51.369835    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9476ce3d_86fb_4e4e_bede_d721052135d4.slice/crio-0a2d9e3048be0165cc654c083d42586edab4d8bd77d486d66f384ba6bca3fa39 WatchSource:0}: Error finding container 0a2d9e3048be0165cc654c083d42586edab4d8bd77d486d66f384ba6bca3fa39: Status 404 returned error can't find the container with id 0a2d9e3048be0165cc654c083d42586edab4d8bd77d486d66f384ba6bca3fa39
Jul 16 21:51:51.409936 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.405492    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqs7" event={"ID":"34db72e0-c3d3-443e-9b0b-d89ea20e4afb","Type":"ContainerStarted","Data":"07d5a2613a9574ea81c4277fa9b60e1283dc071a82f249c26aee7d43048d7f09"}
Jul 16 21:51:51.418394 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.410241    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9" event={"ID":"69e0a640-0edc-4cb6-add3-0e5a25953d8a","Type":"ContainerStarted","Data":"46e46a1479e0c3255689fdb5b034d2f5b3c281a90f2eb0f54044166946354909"}
Jul 16 21:51:51.418394 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.417924    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b" event={"ID":"9476ce3d-86fb-4e4e-bede-d721052135d4","Type":"ContainerStarted","Data":"0a2d9e3048be0165cc654c083d42586edab4d8bd77d486d66f384ba6bca3fa39"}
Jul 16 21:51:51.425211 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.421645    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fdbb98-zhg6n" event={"ID":"940c9e26-3574-4bd7-a47e-c8a8ea13ded9","Type":"ContainerStarted","Data":"e36dbd6c6d1e41e399e6973de4442f84a3371499314ce34b1d555881a12898de"}
Jul 16 21:51:51.457980 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.456313    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr" event={"ID":"c6fc99cf-c316-4484-a451-c51854b1727e","Type":"ContainerStarted","Data":"40a26b24a244904e4a6489148395407f7b3c88c5beb2660cb2e37e2f9b7cf103"}
Jul 16 21:51:51.463010 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.459418    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rpn85" event={"ID":"90027345-b94e-49c5-8d72-0bff02dcdf87","Type":"ContainerStarted","Data":"96447533ddcfff181d0bc76d177f3dedf04e2a7ab74f3be723a978d49f840303"}
Jul 16 21:51:51.463010 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.462762    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-5bc67dcd46-dmksn" event={"ID":"c4a90ba1-3a1e-42b3-91a4-287b04054e2d","Type":"ContainerStarted","Data":"fcda9678037c629b26e0fd7b05e2d57cb5c8a4258f35bd093052bced2540ab54"}
Jul 16 21:51:51.466779 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.466537    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-5bc67dcd46-dmksn"
Jul 16 21:51:51.466779 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.466653    2585 patch_prober.go:28] interesting pod/downloads-5bc67dcd46-dmksn container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.17:8080/\": dial tcp 10.134.0.17:8080: connect: connection refused" start-of-body=
Jul 16 21:51:51.466779 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.466697    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-5bc67dcd46-dmksn" podUID="c4a90ba1-3a1e-42b3-91a4-287b04054e2d" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.17:8080/\": dial tcp 10.134.0.17:8080: connect: connection refused"
Jul 16 21:51:51.596699 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.595573    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-5bc67dcd46-dmksn" podStartSLOduration=3.1072361649999998 podStartE2EDuration="21.595547196s" totalImagesPullingTime="18.488311031s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:51.590395926 +0000 UTC m=+111.394819288" watchObservedRunningTime="2026-07-16 21:51:51.595547196 +0000 UTC m=+111.399970523"
Jul 16 21:51:51.618253 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.617181    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"]
Jul 16 21:51:51.619179 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.619140    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5d47cb5bb8-q4f8k"]
Jul 16 21:51:51.623778 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.622858    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:51.623778 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:51.622972    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d729d56_c88f_49ff_8882_e4e28d58859a.slice/crio-10acbcaea536ffe9e6d80ddb7c4cc1774e78fecfb8176c706bcdf8d40e2df560 WatchSource:0}: Error finding container 10acbcaea536ffe9e6d80ddb7c4cc1774e78fecfb8176c706bcdf8d40e2df560: Status 404 returned error can't find the container with id 10acbcaea536ffe9e6d80ddb7c4cc1774e78fecfb8176c706bcdf8d40e2df560
Jul 16 21:51:51.625403 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:51:51.625190    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod222a7ebc_cdea_429c_9593_3ed72c4061af.slice/crio-df6dc414fb2118140b9b4e79465b244013fce51ce1ebc6f6e34fb601e978f311 WatchSource:0}: Error finding container df6dc414fb2118140b9b4e79465b244013fce51ce1ebc6f6e34fb601e978f311: Status 404 returned error can't find the container with id df6dc414fb2118140b9b4e79465b244013fce51ce1ebc6f6e34fb601e978f311
Jul 16 21:51:51.857221 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:51.857089    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:52.341070 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.341014    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:52.433652 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.433560    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:52.473140 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.472617    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d47cb5bb8-q4f8k" event={"ID":"222a7ebc-cdea-429c-9593-3ed72c4061af","Type":"ContainerStarted","Data":"3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae"}
Jul 16 21:51:52.473140 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.472693    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d47cb5bb8-q4f8k" event={"ID":"222a7ebc-cdea-429c-9593-3ed72c4061af","Type":"ContainerStarted","Data":"df6dc414fb2118140b9b4e79465b244013fce51ce1ebc6f6e34fb601e978f311"}
Jul 16 21:51:52.477783 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.477623    2585 generic.go:356] "Generic (PLEG): container finished" podID="34db72e0-c3d3-443e-9b0b-d89ea20e4afb" containerID="07d5a2613a9574ea81c4277fa9b60e1283dc071a82f249c26aee7d43048d7f09" exitCode=0
Jul 16 21:51:52.477783 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.477719    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqs7" event={"ID":"34db72e0-c3d3-443e-9b0b-d89ea20e4afb","Type":"ContainerDied","Data":"07d5a2613a9574ea81c4277fa9b60e1283dc071a82f249c26aee7d43048d7f09"}
Jul 16 21:51:52.479938 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.479872    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5" event={"ID":"1d729d56-c88f-49ff-8882-e4e28d58859a","Type":"ContainerStarted","Data":"10acbcaea536ffe9e6d80ddb7c4cc1774e78fecfb8176c706bcdf8d40e2df560"}
Jul 16 21:51:52.482986 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.482363    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fdbb98-zhg6n" event={"ID":"940c9e26-3574-4bd7-a47e-c8a8ea13ded9","Type":"ContainerStarted","Data":"c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf"}
Jul 16 21:51:52.491543 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.490809    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr" event={"ID":"c6fc99cf-c316-4484-a451-c51854b1727e","Type":"ContainerStarted","Data":"b26b20e695b8d74fa2c98473dba1deef3891e00f71def6ada02a975931a2ae02"}
Jul 16 21:51:52.491543 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.490889    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr" event={"ID":"c6fc99cf-c316-4484-a451-c51854b1727e","Type":"ContainerStarted","Data":"c83085405a260d3fc9b54f2262766665fdd2488bd8613ec24d015c524f888665"}
Jul 16 21:51:52.496633 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.496594    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-rpn85" event={"ID":"90027345-b94e-49c5-8d72-0bff02dcdf87","Type":"ContainerStarted","Data":"1395b9a759b690e3f8f46f981545d2c51a431a92a6bc541343010a74764e0b1c"}
Jul 16 21:51:52.496633 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.496902    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-rpn85"
Jul 16 21:51:52.508705 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.508437    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-5bc67dcd46-dmksn"
Jul 16 21:51:52.510280 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.510213    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-5d47cb5bb8-q4f8k" podStartSLOduration=2.510191916 podStartE2EDuration="2.510191916s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:52.509207421 +0000 UTC m=+112.313630728" watchObservedRunningTime="2026-07-16 21:51:52.510191916 +0000 UTC m=+112.314615244"
Jul 16 21:51:52.586172 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.586047    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-rpn85" podStartSLOduration=66.142130474 podStartE2EDuration="1m15.586028291s" totalImagesPullingTime="9.443897817s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:52.584599617 +0000 UTC m=+112.389022947" watchObservedRunningTime="2026-07-16 21:51:52.586028291 +0000 UTC m=+112.390451614"
Jul 16 21:51:52.625412 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.625252    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-56c5594674-plpcr" podStartSLOduration=2.579819594 podStartE2EDuration="12.625233726s" totalImagesPullingTime="10.045414132s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:52.623728008 +0000 UTC m=+112.428151335" watchObservedRunningTime="2026-07-16 21:51:52.625233726 +0000 UTC m=+112.429657051"
Jul 16 21:51:52.663040 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:52.661972    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-584fdbb98-zhg6n" podStartSLOduration=10.661950125 podStartE2EDuration="10.661950125s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:52.660725028 +0000 UTC m=+112.465148354" watchObservedRunningTime="2026-07-16 21:51:52.661950125 +0000 UTC m=+112.466373452"
Jul 16 21:51:53.507025 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:53.506956    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqs7" event={"ID":"34db72e0-c3d3-443e-9b0b-d89ea20e4afb","Type":"ContainerStarted","Data":"ea328a6d484aafa297867f8c9ffe4d94c58ec8cc3c6d0d8b255f053c045515b7"}
Jul 16 21:51:53.507025 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:53.507021    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-jvqs7" event={"ID":"34db72e0-c3d3-443e-9b0b-d89ea20e4afb","Type":"ContainerStarted","Data":"0ba2bdd7487c04b0d96dd79756ec3a3dc1ae451b78480894823c75b54fff47bd"}
Jul 16 21:51:53.922603 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:53.922503    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:54.017001 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:54.016348    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:54.115978 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:54.115938    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:55.597052 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:55.596989    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr" podUID="d22bc564-2ae4-44fa-ad34-4a3761415500" containerName="registry" containerID="cri-o://4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45" gracePeriod=30
Jul 16 21:51:55.928329 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:55.928292    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:55.976373 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:55.976316    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-jvqs7" podStartSLOduration=6.218548263 podStartE2EDuration="15.976294238s" totalImagesPullingTime="9.598227151s" totalInitContainerRuntime="159.518824ms" isStatefulPod=true podCreationTimestamp="2026-07-16 21:51:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:53.558366714 +0000 UTC m=+113.362790045" watchObservedRunningTime="2026-07-16 21:51:55.976294238 +0000 UTC m=+115.780717566"
Jul 16 21:51:56.078780 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.078725    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-installation-pull-secrets\") pod \"d22bc564-2ae4-44fa-ad34-4a3761415500\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") "
Jul 16 21:51:56.078940 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.078790    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-image-registry-private-configuration\") pod \"d22bc564-2ae4-44fa-ad34-4a3761415500\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") "
Jul 16 21:51:56.078940 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.078830    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") pod \"d22bc564-2ae4-44fa-ad34-4a3761415500\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") "
Jul 16 21:51:56.078940 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.078847    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-certificates\") pod \"d22bc564-2ae4-44fa-ad34-4a3761415500\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") "
Jul 16 21:51:56.078940 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.078886    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22bc564-2ae4-44fa-ad34-4a3761415500-ca-trust-extracted\") pod \"d22bc564-2ae4-44fa-ad34-4a3761415500\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") "
Jul 16 21:51:56.078940 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.078909    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-bound-sa-token\") pod \"d22bc564-2ae4-44fa-ad34-4a3761415500\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") "
Jul 16 21:51:56.078940 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.078938    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t29xn\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-kube-api-access-t29xn\") pod \"d22bc564-2ae4-44fa-ad34-4a3761415500\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") "
Jul 16 21:51:56.079249 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.078988    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-trusted-ca\") pod \"d22bc564-2ae4-44fa-ad34-4a3761415500\" (UID: \"d22bc564-2ae4-44fa-ad34-4a3761415500\") "
Jul 16 21:51:56.079697 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.079514    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "d22bc564-2ae4-44fa-ad34-4a3761415500" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:56.079697 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.079670    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "d22bc564-2ae4-44fa-ad34-4a3761415500" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:51:56.084169 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.083974    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "d22bc564-2ae4-44fa-ad34-4a3761415500" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:56.084474 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.084421    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "d22bc564-2ae4-44fa-ad34-4a3761415500" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:51:56.085147 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.085092    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "d22bc564-2ae4-44fa-ad34-4a3761415500" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:56.088507 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.088400    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-kube-api-access-t29xn" (OuterVolumeSpecName: "kube-api-access-t29xn") pod "d22bc564-2ae4-44fa-ad34-4a3761415500" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500"). InnerVolumeSpecName "kube-api-access-t29xn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:56.088507 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.088478    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "d22bc564-2ae4-44fa-ad34-4a3761415500" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:51:56.090889 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.090743    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d22bc564-2ae4-44fa-ad34-4a3761415500-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "d22bc564-2ae4-44fa-ad34-4a3761415500" (UID: "d22bc564-2ae4-44fa-ad34-4a3761415500"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:51:56.180484 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.180441    2585 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-image-registry-private-configuration\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:51:56.180484 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.180482    2585 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:51:56.180677 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.180500    2585 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-registry-certificates\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:51:56.180677 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.180516    2585 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/d22bc564-2ae4-44fa-ad34-4a3761415500-ca-trust-extracted\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:51:56.180677 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.180530    2585 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-bound-sa-token\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:51:56.180677 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.180543    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t29xn\" (UniqueName: \"kubernetes.io/projected/d22bc564-2ae4-44fa-ad34-4a3761415500-kube-api-access-t29xn\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:51:56.180677 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.180556    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/d22bc564-2ae4-44fa-ad34-4a3761415500-trusted-ca\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:51:56.180677 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.180569    2585 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/d22bc564-2ae4-44fa-ad34-4a3761415500-installation-pull-secrets\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:51:56.528867 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.528827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b" event={"ID":"9476ce3d-86fb-4e4e-bede-d721052135d4","Type":"ContainerStarted","Data":"51c725f2ac97e550db75c831e60497782d6d87799f27aee94989728b97b95d03"}
Jul 16 21:51:56.529068 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.528995    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:56.530481 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.530443    2585 generic.go:356] "Generic (PLEG): container finished" podID="d22bc564-2ae4-44fa-ad34-4a3761415500" containerID="4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45" exitCode=0
Jul 16 21:51:56.530620 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.530503    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr"
Jul 16 21:51:56.530620 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.530536    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr" event={"ID":"d22bc564-2ae4-44fa-ad34-4a3761415500","Type":"ContainerDied","Data":"4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45"}
Jul 16 21:51:56.530620 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.530595    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-6c9998cdd8-f8hkr" event={"ID":"d22bc564-2ae4-44fa-ad34-4a3761415500","Type":"ContainerDied","Data":"711a72f5a69c6ace1738e6f987da0869da2e6db26aa6b6cd964b8c750780f529"}
Jul 16 21:51:56.530768 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.530633    2585 scope.go:117] "RemoveContainer" containerID="4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45"
Jul 16 21:51:56.540281 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.539903    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9" event={"ID":"69e0a640-0edc-4cb6-add3-0e5a25953d8a","Type":"ContainerStarted","Data":"c7c4c993e7f8964a7898eb1407edb11f5525c4cf25c2563364c66a92d1b07709"}
Jul 16 21:51:56.540281 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.540002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9" event={"ID":"69e0a640-0edc-4cb6-add3-0e5a25953d8a","Type":"ContainerStarted","Data":"50e3d848afc702910e6919b71d50a84a6355571ef96a95d0e387e9bd10e15205"}
Jul 16 21:51:56.540281 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.540046    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9" event={"ID":"69e0a640-0edc-4cb6-add3-0e5a25953d8a","Type":"ContainerStarted","Data":"d7c6feee72292821b678f4f697b396a0f73f169ed41a1881600c7d772a6872e2"}
Jul 16 21:51:56.540529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.540402    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b"
Jul 16 21:51:56.543345 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.543319    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5" event={"ID":"1d729d56-c88f-49ff-8882-e4e28d58859a","Type":"ContainerStarted","Data":"0f1ef34c726a939d0055ea1548ede1cc60d5cb1100075c71cb94671f3f982f9e"}
Jul 16 21:51:56.544635 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.544609    2585 scope.go:117] "RemoveContainer" containerID="4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45"
Jul 16 21:51:56.544993 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:51:56.544967    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45\": container with ID starting with 4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45 not found: ID does not exist" containerID="4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45"
Jul 16 21:51:56.545075 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.545001    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45"} err="failed to get container status \"4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45\": rpc error: code = NotFound desc = could not find container \"4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45\": container with ID starting with 4bb03c5f56a44acc149567fa170452a5017dee526b3bc75546419edec2c9ce45 not found: ID does not exist"
Jul 16 21:51:56.565091 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.565012    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/monitoring-plugin-7b7b7bc8bf-56b6b" podStartSLOduration=7.29983441 podStartE2EDuration="11.564987007s" totalImagesPullingTime="4.265152597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:56.562512823 +0000 UTC m=+116.366936148" watchObservedRunningTime="2026-07-16 21:51:56.564987007 +0000 UTC m=+116.369410334"
Jul 16 21:51:56.593066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.593031    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-6c9998cdd8-f8hkr"]
Jul 16 21:51:56.601507 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.601469    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-6c9998cdd8-f8hkr"]
Jul 16 21:51:56.629515 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.629444    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5" podStartSLOduration=7.6143665689999995 podStartE2EDuration="11.629421695s" totalImagesPullingTime="4.015055126s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:56.628484441 +0000 UTC m=+116.432907768" watchObservedRunningTime="2026-07-16 21:51:56.629421695 +0000 UTC m=+116.433845022"
Jul 16 21:51:56.827825 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:56.827737    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d22bc564-2ae4-44fa-ad34-4a3761415500" path="/var/lib/kubelet/pods/d22bc564-2ae4-44fa-ad34-4a3761415500/volumes"
Jul 16 21:51:58.419412 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.419367    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:58.520131 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.520058    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:58.559181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.559141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9" event={"ID":"69e0a640-0edc-4cb6-add3-0e5a25953d8a","Type":"ContainerStarted","Data":"3af23819d1c5d7ee7992072858d26e376e554a7a8cdd4509fa90d95ea3315a39"}
Jul 16 21:51:58.559382 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.559229    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9" event={"ID":"69e0a640-0edc-4cb6-add3-0e5a25953d8a","Type":"ContainerStarted","Data":"f6ddab2f698fa39cc6b09641bbc94768f98516f1f24750fa63a040457d4abfd6"}
Jul 16 21:51:58.559382 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.559273    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9" event={"ID":"69e0a640-0edc-4cb6-add3-0e5a25953d8a","Type":"ContainerStarted","Data":"15b3103e9ee79e84c15612570af64abc54cbabb873c60affbb586ceee0c478ca"}
Jul 16 21:51:58.559675 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.559653    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:51:58.608308 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.608264    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:58.618188 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.617239    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9" podStartSLOduration=9.823233315 podStartE2EDuration="15.617217692s" totalImagesPullingTime="5.793984377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:51:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:51:58.606859748 +0000 UTC m=+118.411283073" watchObservedRunningTime="2026-07-16 21:51:58.617217692 +0000 UTC m=+118.421640997"
Jul 16 21:51:58.948985 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:58.948945    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:59.045721 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:59.045671    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:59.205030 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:59.204948    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:59.309759 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:59.309668    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:51:59.959985 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:51:59.959941    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:00.023886 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:00.023837    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:00.342062 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:00.342020    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:00.443963 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:00.443930    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:00.516599 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:00.516558    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:01.320467 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.320425    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:52:01.320965 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.320490    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:52:01.325450 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.325425    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:52:01.366612 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.366578    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:01.555255 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.555215    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:01.577448 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.577369    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:52:01.656339 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.656305    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584fdbb98-zhg6n"]
Jul 16 21:52:01.766747 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.766705    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:01.872903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:01.872802    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:02.144044 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:02.143948    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:02.245148 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:02.245112    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:02.435023 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:02.434935    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:52:02.511852 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:02.511818    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-rpn85"
Jul 16 21:52:04.571850 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:04.571816    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-86656b897-m6jt9"
Jul 16 21:52:05.516186 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:05.516150    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:52:05.516361 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:05.516211    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:52:08.322885 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:08.322835    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:08.424846 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:08.424803    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:08.611037 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:08.610951    2585 generic.go:356] "Generic (PLEG): container finished" podID="2a5960e6-9c85-4d5a-947e-c4075b287904" containerID="38c46b18011f5c0159dee2cd16a3d277e84ca334df29e4a1cfbd35d03c7f5909" exitCode=0
Jul 16 21:52:08.611037 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:08.611022    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-4bfcb" event={"ID":"2a5960e6-9c85-4d5a-947e-c4075b287904","Type":"ContainerDied","Data":"38c46b18011f5c0159dee2cd16a3d277e84ca334df29e4a1cfbd35d03c7f5909"}
Jul 16 21:52:08.611403 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:08.611387    2585 scope.go:117] "RemoveContainer" containerID="38c46b18011f5c0159dee2cd16a3d277e84ca334df29e4a1cfbd35d03c7f5909"
Jul 16 21:52:09.616641 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:09.616605    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-555d9d7885-4bfcb" event={"ID":"2a5960e6-9c85-4d5a-947e-c4075b287904","Type":"ContainerStarted","Data":"6c70187097e3d7c4a597cc8fe15efd1ee74bca1d4bc6a2fad298ec7165def2cf"}
Jul 16 21:52:10.523932 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:10.523890    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:52:10.526388 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:10.526359    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3b5ea743-a8b0-4e62-9217-8215b918bee1-metrics-certs\") pod \"network-metrics-daemon-l9sj7\" (UID: \"3b5ea743-a8b0-4e62-9217-8215b918bee1\") " pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:52:10.660191 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:10.660158    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-xshkd\""
Jul 16 21:52:10.662179 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:10.662155    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-l9sj7"
Jul 16 21:52:10.827659 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:52:10.827630    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b5ea743_a8b0_4e62_9217_8215b918bee1.slice/crio-dcab45002641bc349c64805803f4dce29b681390304655bdfc0f87b275c81de7 WatchSource:0}: Error finding container dcab45002641bc349c64805803f4dce29b681390304655bdfc0f87b275c81de7: Status 404 returned error can't find the container with id dcab45002641bc349c64805803f4dce29b681390304655bdfc0f87b275c81de7
Jul 16 21:52:10.828982 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:10.828937    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-l9sj7"]
Jul 16 21:52:11.047048 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:11.047012    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:11.147676 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:11.147579    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:11.626464 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:11.626427    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9sj7" event={"ID":"3b5ea743-a8b0-4e62-9217-8215b918bee1","Type":"ContainerStarted","Data":"dcab45002641bc349c64805803f4dce29b681390304655bdfc0f87b275c81de7"}
Jul 16 21:52:11.709888 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:11.709838    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:11.783901 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:11.783857    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:12.632544 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:12.632485    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9sj7" event={"ID":"3b5ea743-a8b0-4e62-9217-8215b918bee1","Type":"ContainerStarted","Data":"a4564edbd426f9f418928ff819e9476efd642f73d932c3bf49d67767ea05305b"}
Jul 16 21:52:12.632719 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:12.632590    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-l9sj7" event={"ID":"3b5ea743-a8b0-4e62-9217-8215b918bee1","Type":"ContainerStarted","Data":"1c372313f2b5dbf5e27374a72e79f7dea1195662198cb1d6af9a7d83ece6d7d2"}
Jul 16 21:52:12.664900 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:12.664840    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-l9sj7" podStartSLOduration=130.252056055 podStartE2EDuration="2m11.664805479s" totalImagesPullingTime="1.412749424s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:50:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:52:12.664767661 +0000 UTC m=+132.469190988" watchObservedRunningTime="2026-07-16 21:52:12.664805479 +0000 UTC m=+132.469228805"
Jul 16 21:52:12.813437 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:12.813341    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:12.907943 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:12.907903    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:12.997202 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:12.997167    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:15.450300 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:15.450258    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:15.553223 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:15.553177    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:15.953596 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:15.953546    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-784dfc5755-dtzzx" podUID="a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" containerName="console" containerID="cri-o://c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c" gracePeriod=15
Jul 16 21:52:16.232711 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.232687    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:52:16.376803 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.376769    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-oauth-config\") pod \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") "
Jul 16 21:52:16.376803 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.376808    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-oauth-serving-cert\") pod \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") "
Jul 16 21:52:16.377066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.376849    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-config\") pod \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") "
Jul 16 21:52:16.377066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.376882    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-service-ca\") pod \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") "
Jul 16 21:52:16.377066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.376950    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pwwn6\" (UniqueName: \"kubernetes.io/projected/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-kube-api-access-pwwn6\") pod \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") "
Jul 16 21:52:16.377066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.377046    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-serving-cert\") pod \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\" (UID: \"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2\") "
Jul 16 21:52:16.377375 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.377344    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-config" (OuterVolumeSpecName: "console-config") pod "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" (UID: "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:16.377375 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.377356    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-service-ca" (OuterVolumeSpecName: "service-ca") pod "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" (UID: "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:16.377375 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.377361    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" (UID: "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:16.379565 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.379534    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" (UID: "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:16.379662 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.379557    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" (UID: "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:16.379662 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.379575    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-kube-api-access-pwwn6" (OuterVolumeSpecName: "kube-api-access-pwwn6") pod "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" (UID: "a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2"). InnerVolumeSpecName "kube-api-access-pwwn6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:16.478165 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.478125    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-service-ca\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:16.478165 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.478159    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwwn6\" (UniqueName: \"kubernetes.io/projected/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-kube-api-access-pwwn6\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:16.478165 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.478170    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:16.478645 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.478179    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-oauth-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:16.478645 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.478188    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-oauth-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:16.478645 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.478195    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2-console-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:16.650833 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.650745    2585 generic.go:356] "Generic (PLEG): container finished" podID="a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" containerID="c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c" exitCode=0
Jul 16 21:52:16.650833 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.650793    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-784dfc5755-dtzzx"
Jul 16 21:52:16.652488 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.650954    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784dfc5755-dtzzx" event={"ID":"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2","Type":"ContainerDied","Data":"c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c"}
Jul 16 21:52:16.652488 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.651320    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-784dfc5755-dtzzx" event={"ID":"a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2","Type":"ContainerDied","Data":"c85d0ad0cf3cb75ddd415200cd3c1fba139adfd94dea8718a0bd760033f43c4f"}
Jul 16 21:52:16.652488 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.651394    2585 scope.go:117] "RemoveContainer" containerID="c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c"
Jul 16 21:52:16.665850 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.665827    2585 scope.go:117] "RemoveContainer" containerID="c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c"
Jul 16 21:52:16.666207 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:52:16.666177    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c\": container with ID starting with c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c not found: ID does not exist" containerID="c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c"
Jul 16 21:52:16.666313 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.666214    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c"} err="failed to get container status \"c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c\": rpc error: code = NotFound desc = could not find container \"c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c\": container with ID starting with c743ddca1f15de33c9c3c97bdbb87f82b9cfd55e957380b5280d19bd1edf4c1c not found: ID does not exist"
Jul 16 21:52:16.698838 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.698800    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-784dfc5755-dtzzx"]
Jul 16 21:52:16.707492 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.707462    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-784dfc5755-dtzzx"]
Jul 16 21:52:16.830071 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:16.830031    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" path="/var/lib/kubelet/pods/a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2/volumes"
Jul 16 21:52:25.522282 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:25.522245    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:52:25.526345 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:25.526319    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-69bf49fd5f-s7lk5"
Jul 16 21:52:26.678779 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:26.678706    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-584fdbb98-zhg6n" podUID="940c9e26-3574-4bd7-a47e-c8a8ea13ded9" containerName="console" containerID="cri-o://c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf" gracePeriod=15
Jul 16 21:52:26.967751 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:26.967701    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:52:27.079572 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.079529    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-service-ca\") pod \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") "
Jul 16 21:52:27.079572 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.079578    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-oauth-serving-cert\") pod \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") "
Jul 16 21:52:27.079820 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.079617    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvkmk\" (UniqueName: \"kubernetes.io/projected/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-kube-api-access-hvkmk\") pod \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") "
Jul 16 21:52:27.079820 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.079686    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-config\") pod \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") "
Jul 16 21:52:27.079935 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.079834    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-serving-cert\") pod \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") "
Jul 16 21:52:27.079935 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.079895    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-trusted-ca-bundle\") pod \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") "
Jul 16 21:52:27.079935 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.079924    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-oauth-config\") pod \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\" (UID: \"940c9e26-3574-4bd7-a47e-c8a8ea13ded9\") "
Jul 16 21:52:27.080070 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.080019    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "940c9e26-3574-4bd7-a47e-c8a8ea13ded9" (UID: "940c9e26-3574-4bd7-a47e-c8a8ea13ded9"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:27.080070 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.080034    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-service-ca" (OuterVolumeSpecName: "service-ca") pod "940c9e26-3574-4bd7-a47e-c8a8ea13ded9" (UID: "940c9e26-3574-4bd7-a47e-c8a8ea13ded9"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:27.080070 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.080043    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-config" (OuterVolumeSpecName: "console-config") pod "940c9e26-3574-4bd7-a47e-c8a8ea13ded9" (UID: "940c9e26-3574-4bd7-a47e-c8a8ea13ded9"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:27.080304 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.080282    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "940c9e26-3574-4bd7-a47e-c8a8ea13ded9" (UID: "940c9e26-3574-4bd7-a47e-c8a8ea13ded9"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:52:27.080428 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.080410    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:27.080486 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.080434    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-trusted-ca-bundle\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:27.080486 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.080450    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-service-ca\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:27.080486 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.080463    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-oauth-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:27.082143 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.082122    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "940c9e26-3574-4bd7-a47e-c8a8ea13ded9" (UID: "940c9e26-3574-4bd7-a47e-c8a8ea13ded9"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:27.082469 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.082440    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-kube-api-access-hvkmk" (OuterVolumeSpecName: "kube-api-access-hvkmk") pod "940c9e26-3574-4bd7-a47e-c8a8ea13ded9" (UID: "940c9e26-3574-4bd7-a47e-c8a8ea13ded9"). InnerVolumeSpecName "kube-api-access-hvkmk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:52:27.082572 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.082538    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "940c9e26-3574-4bd7-a47e-c8a8ea13ded9" (UID: "940c9e26-3574-4bd7-a47e-c8a8ea13ded9"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:52:27.181030 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.180994    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:27.181030 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.181027    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-console-oauth-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:27.181030 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.181038    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvkmk\" (UniqueName: \"kubernetes.io/projected/940c9e26-3574-4bd7-a47e-c8a8ea13ded9-kube-api-access-hvkmk\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:52:27.711236 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.711197    2585 generic.go:356] "Generic (PLEG): container finished" podID="940c9e26-3574-4bd7-a47e-c8a8ea13ded9" containerID="c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf" exitCode=0
Jul 16 21:52:27.711236 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.711238    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-584fdbb98-zhg6n"
Jul 16 21:52:27.711757 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.711268    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fdbb98-zhg6n" event={"ID":"940c9e26-3574-4bd7-a47e-c8a8ea13ded9","Type":"ContainerDied","Data":"c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf"}
Jul 16 21:52:27.711757 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.711355    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-584fdbb98-zhg6n" event={"ID":"940c9e26-3574-4bd7-a47e-c8a8ea13ded9","Type":"ContainerDied","Data":"e36dbd6c6d1e41e399e6973de4442f84a3371499314ce34b1d555881a12898de"}
Jul 16 21:52:27.711757 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.711388    2585 scope.go:117] "RemoveContainer" containerID="c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf"
Jul 16 21:52:27.720490 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.720468    2585 scope.go:117] "RemoveContainer" containerID="c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf"
Jul 16 21:52:27.720798 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:52:27.720777    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf\": container with ID starting with c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf not found: ID does not exist" containerID="c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf"
Jul 16 21:52:27.720866 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.720808    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf"} err="failed to get container status \"c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf\": rpc error: code = NotFound desc = could not find container \"c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf\": container with ID starting with c941f83030a784b77cc9cef89d1517f24f5d5c20f7cf35a37167c8331ef6acbf not found: ID does not exist"
Jul 16 21:52:27.731698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.731660    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-584fdbb98-zhg6n"]
Jul 16 21:52:27.735317 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:27.735288    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-584fdbb98-zhg6n"]
Jul 16 21:52:28.825897 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:28.825856    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="940c9e26-3574-4bd7-a47e-c8a8ea13ded9" path="/var/lib/kubelet/pods/940c9e26-3574-4bd7-a47e-c8a8ea13ded9/volumes"
Jul 16 21:52:30.478229 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:30.478192    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:30.598567 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:30.598527    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:59.876445 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:59.876244    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:52:59.974513 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:52:59.974470    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:03.195142 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195090    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195407    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="940c9e26-3574-4bd7-a47e-c8a8ea13ded9" containerName="console"
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195418    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="940c9e26-3574-4bd7-a47e-c8a8ea13ded9" containerName="console"
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195428    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" containerName="console"
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195434    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" containerName="console"
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195446    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d22bc564-2ae4-44fa-ad34-4a3761415500" containerName="registry"
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195452    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d22bc564-2ae4-44fa-ad34-4a3761415500" containerName="registry"
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195509    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1eb9826-cdfe-40ac-83bb-a7c40c5a3cf2" containerName="console"
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195519    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="940c9e26-3574-4bd7-a47e-c8a8ea13ded9" containerName="console"
Jul 16 21:53:03.195529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.195524    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d22bc564-2ae4-44fa-ad34-4a3761415500" containerName="registry"
Jul 16 21:53:03.200594 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.200568    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.203136 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.203081    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 16 21:53:03.203301 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.203151    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 16 21:53:03.203301 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.203242    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 16 21:53:03.203415 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.203375    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 16 21:53:03.203566 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.203552    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 16 21:53:03.203624 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.203571    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 16 21:53:03.203624 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.203578    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 16 21:53:03.203876 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.203859    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 16 21:53:03.210757 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.208739    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-44bwl\""
Jul 16 21:53:03.212472 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.212442    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 16 21:53:03.220085 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.220042    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:53:03.298016 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.297968    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298016 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298016    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1dc36b25-f6be-4d30-8669-63324253467c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298272 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298091    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dc36b25-f6be-4d30-8669-63324253467c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298272 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298185    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc36b25-f6be-4d30-8669-63324253467c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298272 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298240    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298390 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298273    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298390 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298355    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1dc36b25-f6be-4d30-8669-63324253467c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298465 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298391    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1dc36b25-f6be-4d30-8669-63324253467c-config-out\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298465 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298423    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298465 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298440    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-web-config\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298602 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298464    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298602 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298480    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2ll5\" (UniqueName: \"kubernetes.io/projected/1dc36b25-f6be-4d30-8669-63324253467c-kube-api-access-q2ll5\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.298602 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.298575    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.399633 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.399589    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc36b25-f6be-4d30-8669-63324253467c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.399851 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.399655    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.399851 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.399682    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.399851 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.399705    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1dc36b25-f6be-4d30-8669-63324253467c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.399851 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.399730    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1dc36b25-f6be-4d30-8669-63324253467c-config-out\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.399851 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.399755    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.399851 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.399777    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-web-config\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.400258 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.400228    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/1dc36b25-f6be-4d30-8669-63324253467c-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.400343 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.400302    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.400397 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.400340    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2ll5\" (UniqueName: \"kubernetes.io/projected/1dc36b25-f6be-4d30-8669-63324253467c-kube-api-access-q2ll5\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.400447 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.400401    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.400496 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.400443    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.400496 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.400472    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1dc36b25-f6be-4d30-8669-63324253467c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.400596 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.400500    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dc36b25-f6be-4d30-8669-63324253467c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.400888 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.400863    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/1dc36b25-f6be-4d30-8669-63324253467c-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.401279 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.401252    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/1dc36b25-f6be-4d30-8669-63324253467c-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.402965 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.402919    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.403074 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.403025    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.403074 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.403062    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.403219 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.403125    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/1dc36b25-f6be-4d30-8669-63324253467c-config-out\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.403514 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.403487    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-web-config\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.403995 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.403971    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/1dc36b25-f6be-4d30-8669-63324253467c-tls-assets\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.404071 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.404003    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-config-volume\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.404634 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.404617    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.404949 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.404933    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/1dc36b25-f6be-4d30-8669-63324253467c-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.410780 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.410758    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2ll5\" (UniqueName: \"kubernetes.io/projected/1dc36b25-f6be-4d30-8669-63324253467c-kube-api-access-q2ll5\") pod \"alertmanager-main-0\" (UID: \"1dc36b25-f6be-4d30-8669-63324253467c\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.516737 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.516697    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 16 21:53:03.667743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.667594    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 16 21:53:03.670619 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:53:03.670578    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1dc36b25_f6be_4d30_8669_63324253467c.slice/crio-8b25bccc9a45b786abab91a620f8db5f2744b930b76c2499d5977c6ca2a443fe WatchSource:0}: Error finding container 8b25bccc9a45b786abab91a620f8db5f2744b930b76c2499d5977c6ca2a443fe: Status 404 returned error can't find the container with id 8b25bccc9a45b786abab91a620f8db5f2744b930b76c2499d5977c6ca2a443fe
Jul 16 21:53:03.862749 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:03.862658    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1dc36b25-f6be-4d30-8669-63324253467c","Type":"ContainerStarted","Data":"8b25bccc9a45b786abab91a620f8db5f2744b930b76c2499d5977c6ca2a443fe"}
Jul 16 21:53:04.867901 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.867856    2585 generic.go:356] "Generic (PLEG): container finished" podID="1dc36b25-f6be-4d30-8669-63324253467c" containerID="f56289ce76fd5dfcfba27e5a14b51d487c0328c4c38af2b515ba827d989244cd" exitCode=0
Jul 16 21:53:04.868413 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.867947    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1dc36b25-f6be-4d30-8669-63324253467c","Type":"ContainerDied","Data":"f56289ce76fd5dfcfba27e5a14b51d487c0328c4c38af2b515ba827d989244cd"}
Jul 16 21:53:04.984619 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.984579    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"]
Jul 16 21:53:04.988372 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.988340    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:04.991584 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.991540    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Jul 16 21:53:04.992217 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.991932    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-t7kh9\""
Jul 16 21:53:04.992217 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.991974    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Jul 16 21:53:04.992217 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.992039    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Jul 16 21:53:04.992217 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.992110    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Jul 16 21:53:04.992217 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.991974    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Jul 16 21:53:04.998411 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:04.998373    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Jul 16 21:53:05.004135 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.004068    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"]
Jul 16 21:53:05.121273 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.121225    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-serving-certs-ca-bundle\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.121515 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.121277    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw5fc\" (UniqueName: \"kubernetes.io/projected/df481018-75a5-4b01-9797-ee333a342de0-kube-api-access-dw5fc\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.121515 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.121401    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.121515 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.121461    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-telemeter-client-tls\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.121515 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.121495    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-secret-telemeter-client\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.121687 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.121524    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-federate-client-tls\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.121687 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.121582    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.121687 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.121639    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-metrics-client-ca\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.222341 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.222297    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-metrics-client-ca\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.222651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.222416    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-serving-certs-ca-bundle\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.222651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.222443    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw5fc\" (UniqueName: \"kubernetes.io/projected/df481018-75a5-4b01-9797-ee333a342de0-kube-api-access-dw5fc\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.222651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.222478    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.222651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.222506    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-telemeter-client-tls\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.222651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.222537    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-secret-telemeter-client\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.222651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.222566    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-federate-client-tls\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.222651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.222623    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.223253 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.223224    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-metrics-client-ca\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.223365 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.223234    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-serving-certs-ca-bundle\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.223701 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.223672    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/df481018-75a5-4b01-9797-ee333a342de0-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.225559 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.225479    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-federate-client-tls\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.225559 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.225495    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.227814 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.225866    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-secret-telemeter-client\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.228251 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.228230    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/df481018-75a5-4b01-9797-ee333a342de0-telemeter-client-tls\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.233842 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.233787    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw5fc\" (UniqueName: \"kubernetes.io/projected/df481018-75a5-4b01-9797-ee333a342de0-kube-api-access-dw5fc\") pod \"telemeter-client-7b87bbcfcf-b69xm\" (UID: \"df481018-75a5-4b01-9797-ee333a342de0\") " pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.303684 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.303590    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"
Jul 16 21:53:05.481965 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.481912    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm"]
Jul 16 21:53:05.485492 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:53:05.485460    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf481018_75a5_4b01_9797_ee333a342de0.slice/crio-3aa086619d0eebc882553f37dacbbf4be3bfe440733eab4aa5d09e23a42da69e WatchSource:0}: Error finding container 3aa086619d0eebc882553f37dacbbf4be3bfe440733eab4aa5d09e23a42da69e: Status 404 returned error can't find the container with id 3aa086619d0eebc882553f37dacbbf4be3bfe440733eab4aa5d09e23a42da69e
Jul 16 21:53:05.878167 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:05.878121    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm" event={"ID":"df481018-75a5-4b01-9797-ee333a342de0","Type":"ContainerStarted","Data":"3aa086619d0eebc882553f37dacbbf4be3bfe440733eab4aa5d09e23a42da69e"}
Jul 16 21:53:07.888893 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.888853    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm" event={"ID":"df481018-75a5-4b01-9797-ee333a342de0","Type":"ContainerStarted","Data":"76a93415a780ed81c62d891510f66d34a1f361e9f7a5069acb776287f982edd5"}
Jul 16 21:53:07.889370 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.888926    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm" event={"ID":"df481018-75a5-4b01-9797-ee333a342de0","Type":"ContainerStarted","Data":"dbb179157184167d359052b3007f3dc4929d2236c2f1c503c207cd60dd201e75"}
Jul 16 21:53:07.889370 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.888960    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm" event={"ID":"df481018-75a5-4b01-9797-ee333a342de0","Type":"ContainerStarted","Data":"6248593797ca1d87dd497eb1743a713ab70340d5084ebbc1253681f4c4b293e0"}
Jul 16 21:53:07.891991 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.891961    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1dc36b25-f6be-4d30-8669-63324253467c","Type":"ContainerStarted","Data":"57c3c1dcea39ec6414467252a57eaaae6f2e23d51d2280de7d7b576c6dcd5540"}
Jul 16 21:53:07.892148 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.892031    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1dc36b25-f6be-4d30-8669-63324253467c","Type":"ContainerStarted","Data":"27b879054e19316effe39f0c98b566f0073669b98a286f54b8c446166494b89e"}
Jul 16 21:53:07.892148 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.892058    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1dc36b25-f6be-4d30-8669-63324253467c","Type":"ContainerStarted","Data":"4cbc77ebe011e5d2cf412c70fb310133108b158784fc75695e659d79c4a10131"}
Jul 16 21:53:07.892148 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.892088    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1dc36b25-f6be-4d30-8669-63324253467c","Type":"ContainerStarted","Data":"cdae3b0b810575c6f4130158bff7c057352073c06b7308fe4498e49c108cb15d"}
Jul 16 21:53:07.892148 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.892138    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1dc36b25-f6be-4d30-8669-63324253467c","Type":"ContainerStarted","Data":"57a0e7007a752f6ac814afb69bbd34697efa74b1e763dcc0f80539b8c8645ee8"}
Jul 16 21:53:07.892279 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.892166    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"1dc36b25-f6be-4d30-8669-63324253467c","Type":"ContainerStarted","Data":"726c67fe25589772e8a71c36ae89e68eb6cb8039460ddc6d3944c7477d017524"}
Jul 16 21:53:07.922217 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.922160    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7b87bbcfcf-b69xm" podStartSLOduration=2.314282829 podStartE2EDuration="3.922140805s" totalImagesPullingTime="1.607857976s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:07.918225703 +0000 UTC m=+187.722649029" watchObservedRunningTime="2026-07-16 21:53:07.922140805 +0000 UTC m=+187.726564128"
Jul 16 21:53:07.953607 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:07.953546    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=1.720228756 podStartE2EDuration="4.953526411s" totalImagesPullingTime="3.212096112s" totalInitContainerRuntime="21.201543ms" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:03 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:07.951439715 +0000 UTC m=+187.755863081" watchObservedRunningTime="2026-07-16 21:53:07.953526411 +0000 UTC m=+187.757949736"
Jul 16 21:53:08.776395 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.776357    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6477dbdbf-5qp9f"]
Jul 16 21:53:08.780256 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.780228    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.793071 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.793041    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6477dbdbf-5qp9f"]
Jul 16 21:53:08.839961 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.839913    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:08.860262 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.860211    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-trusted-ca-bundle\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.860262 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.860268    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-oauth-serving-cert\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.860486 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.860302    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-oauth-config\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.860486 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.860360    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-console-config\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.860486 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.860401    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtc4t\" (UniqueName: \"kubernetes.io/projected/481531eb-3e1e-463a-a192-8d079f97372d-kube-api-access-wtc4t\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.860580 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.860488    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-serving-cert\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.860580 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.860504    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-service-ca\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.912147 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.912090    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:08.961306 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.961267    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-oauth-config\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.961505 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.961313    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-console-config\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.961505 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.961347    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtc4t\" (UniqueName: \"kubernetes.io/projected/481531eb-3e1e-463a-a192-8d079f97372d-kube-api-access-wtc4t\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.961505 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.961477    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-serving-cert\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.961505 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.961499    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-service-ca\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.961727 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.961544    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-trusted-ca-bundle\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.961727 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.961578    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-oauth-serving-cert\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.962242 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.962209    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-console-config\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.962491 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.962471    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-trusted-ca-bundle\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.962579 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.962476    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-service-ca\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.962579 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.962563    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-oauth-serving-cert\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.964182 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.964158    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-serving-cert\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.964274 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.964186    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-oauth-config\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.970045 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.970026    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtc4t\" (UniqueName: \"kubernetes.io/projected/481531eb-3e1e-463a-a192-8d079f97372d-kube-api-access-wtc4t\") pod \"console-6477dbdbf-5qp9f\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") " pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:08.975072 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:08.975038    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:09.091422 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:09.091312    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:09.256159 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:09.256120    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6477dbdbf-5qp9f"]
Jul 16 21:53:09.259407 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:53:09.259368    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod481531eb_3e1e_463a_a192_8d079f97372d.slice/crio-ab86803c6430bb93c9f44453a5f8cb71c695dca566c6d8fb4fa00649e21c5e60 WatchSource:0}: Error finding container ab86803c6430bb93c9f44453a5f8cb71c695dca566c6d8fb4fa00649e21c5e60: Status 404 returned error can't find the container with id ab86803c6430bb93c9f44453a5f8cb71c695dca566c6d8fb4fa00649e21c5e60
Jul 16 21:53:09.605471 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:09.605428    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:09.669738 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:09.669687    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:09.901558 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:09.901462    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477dbdbf-5qp9f" event={"ID":"481531eb-3e1e-463a-a192-8d079f97372d","Type":"ContainerStarted","Data":"b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e"}
Jul 16 21:53:09.901558 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:09.901520    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477dbdbf-5qp9f" event={"ID":"481531eb-3e1e-463a-a192-8d079f97372d","Type":"ContainerStarted","Data":"ab86803c6430bb93c9f44453a5f8cb71c695dca566c6d8fb4fa00649e21c5e60"}
Jul 16 21:53:09.919952 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:09.919892    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6477dbdbf-5qp9f" podStartSLOduration=1.91987385 podStartE2EDuration="1.91987385s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:53:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:53:09.919208886 +0000 UTC m=+189.723632212" watchObservedRunningTime="2026-07-16 21:53:09.91987385 +0000 UTC m=+189.724297175"
Jul 16 21:53:19.091610 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:19.091550    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:19.092079 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:19.091652    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:19.096658 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:19.096630    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:19.950861 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:19.950830    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:53:20.000006 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:19.999965    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d47cb5bb8-q4f8k"]
Jul 16 21:53:20.209846 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:20.209751    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:20.307316 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:20.307277    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:39.601154 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:39.601085    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:39.668225 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:39.668185    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:40.216166 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:40.216128    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:40.275587 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:40.275552    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:44.068993 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:44.068949    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:53:45.025579 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.025534    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-5d47cb5bb8-q4f8k" podUID="222a7ebc-cdea-429c-9593-3ed72c4061af" containerName="console" containerID="cri-o://3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae" gracePeriod=15
Jul 16 21:53:45.281774 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.281696    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:53:45.395767 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.395723    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-trusted-ca-bundle\") pod \"222a7ebc-cdea-429c-9593-3ed72c4061af\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") "
Jul 16 21:53:45.395959 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.395787    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-console-config\") pod \"222a7ebc-cdea-429c-9593-3ed72c4061af\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") "
Jul 16 21:53:45.395959 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.395816    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-service-ca\") pod \"222a7ebc-cdea-429c-9593-3ed72c4061af\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") "
Jul 16 21:53:45.395959 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.395905    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-oauth-serving-cert\") pod \"222a7ebc-cdea-429c-9593-3ed72c4061af\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") "
Jul 16 21:53:45.396237 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.395962    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-oauth-config\") pod \"222a7ebc-cdea-429c-9593-3ed72c4061af\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") "
Jul 16 21:53:45.396237 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396012    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-serving-cert\") pod \"222a7ebc-cdea-429c-9593-3ed72c4061af\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") "
Jul 16 21:53:45.396237 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396039    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2vpcg\" (UniqueName: \"kubernetes.io/projected/222a7ebc-cdea-429c-9593-3ed72c4061af-kube-api-access-2vpcg\") pod \"222a7ebc-cdea-429c-9593-3ed72c4061af\" (UID: \"222a7ebc-cdea-429c-9593-3ed72c4061af\") "
Jul 16 21:53:45.396399 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396282    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "222a7ebc-cdea-429c-9593-3ed72c4061af" (UID: "222a7ebc-cdea-429c-9593-3ed72c4061af"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:53:45.396399 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396305    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-console-config" (OuterVolumeSpecName: "console-config") pod "222a7ebc-cdea-429c-9593-3ed72c4061af" (UID: "222a7ebc-cdea-429c-9593-3ed72c4061af"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:53:45.396399 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396322    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-service-ca" (OuterVolumeSpecName: "service-ca") pod "222a7ebc-cdea-429c-9593-3ed72c4061af" (UID: "222a7ebc-cdea-429c-9593-3ed72c4061af"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:53:45.396518 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396414    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "222a7ebc-cdea-429c-9593-3ed72c4061af" (UID: "222a7ebc-cdea-429c-9593-3ed72c4061af"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:53:45.396518 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396431    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-trusted-ca-bundle\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:53:45.396518 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396455    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-console-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:53:45.396518 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.396473    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-service-ca\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:53:45.398275 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.398246    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/222a7ebc-cdea-429c-9593-3ed72c4061af-kube-api-access-2vpcg" (OuterVolumeSpecName: "kube-api-access-2vpcg") pod "222a7ebc-cdea-429c-9593-3ed72c4061af" (UID: "222a7ebc-cdea-429c-9593-3ed72c4061af"). InnerVolumeSpecName "kube-api-access-2vpcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:53:45.398275 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.398251    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "222a7ebc-cdea-429c-9593-3ed72c4061af" (UID: "222a7ebc-cdea-429c-9593-3ed72c4061af"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:53:45.398437 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.398384    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "222a7ebc-cdea-429c-9593-3ed72c4061af" (UID: "222a7ebc-cdea-429c-9593-3ed72c4061af"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:53:45.497937 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.497892    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/222a7ebc-cdea-429c-9593-3ed72c4061af-oauth-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:53:45.497937 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.497926    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-oauth-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:53:45.497937 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.497936    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/222a7ebc-cdea-429c-9593-3ed72c4061af-console-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:53:45.497937 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:45.497945    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vpcg\" (UniqueName: \"kubernetes.io/projected/222a7ebc-cdea-429c-9593-3ed72c4061af-kube-api-access-2vpcg\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:53:46.059422 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.059387    2585 generic.go:356] "Generic (PLEG): container finished" podID="222a7ebc-cdea-429c-9593-3ed72c4061af" containerID="3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae" exitCode=0
Jul 16 21:53:46.059422 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.059427    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d47cb5bb8-q4f8k" event={"ID":"222a7ebc-cdea-429c-9593-3ed72c4061af","Type":"ContainerDied","Data":"3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae"}
Jul 16 21:53:46.059422 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.059435    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5d47cb5bb8-q4f8k"
Jul 16 21:53:46.059683 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.059472    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5d47cb5bb8-q4f8k" event={"ID":"222a7ebc-cdea-429c-9593-3ed72c4061af","Type":"ContainerDied","Data":"df6dc414fb2118140b9b4e79465b244013fce51ce1ebc6f6e34fb601e978f311"}
Jul 16 21:53:46.059683 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.059492    2585 scope.go:117] "RemoveContainer" containerID="3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae"
Jul 16 21:53:46.068529 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.068508    2585 scope.go:117] "RemoveContainer" containerID="3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae"
Jul 16 21:53:46.068840 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:53:46.068818    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae\": container with ID starting with 3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae not found: ID does not exist" containerID="3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae"
Jul 16 21:53:46.068904 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.068854    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae"} err="failed to get container status \"3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae\": rpc error: code = NotFound desc = could not find container \"3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae\": container with ID starting with 3467c8bfb8192e34039ea6021b7d8f04ec2db52a34273c040b988bf567a5aaae not found: ID does not exist"
Jul 16 21:53:46.085313 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.085280    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5d47cb5bb8-q4f8k"]
Jul 16 21:53:46.090109 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.090069    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5d47cb5bb8-q4f8k"]
Jul 16 21:53:46.826214 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:53:46.826174    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="222a7ebc-cdea-429c-9593-3ed72c4061af" path="/var/lib/kubelet/pods/222a7ebc-cdea-429c-9593-3ed72c4061af/volumes"
Jul 16 21:54:15.195040 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:15.195001    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:15.341357 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:15.341316    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:26.165613 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.165520    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-55f599b986-6ddpp"]
Jul 16 21:54:26.166067 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.166051    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="222a7ebc-cdea-429c-9593-3ed72c4061af" containerName="console"
Jul 16 21:54:26.166135 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.166073    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="222a7ebc-cdea-429c-9593-3ed72c4061af" containerName="console"
Jul 16 21:54:26.166244 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.166232    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="222a7ebc-cdea-429c-9593-3ed72c4061af" containerName="console"
Jul 16 21:54:26.169518 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.169498    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.179022 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.178993    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f599b986-6ddpp"]
Jul 16 21:54:26.241222 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.241183    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:26.275397 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.275356    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-service-ca\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.275397 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.275400    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-trusted-ca-bundle\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.275608 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.275422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-serving-cert\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.275608 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.275542    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-oauth-serving-cert\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.275608 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.275583    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hc2zg\" (UniqueName: \"kubernetes.io/projected/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-kube-api-access-hc2zg\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.275608 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.275601    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-config\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.275731 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.275625    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-oauth-config\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.307954 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.307920    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:26.376998 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.376944    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-service-ca\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.376998 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377004    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-trusted-ca-bundle\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.377314 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377034    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-serving-cert\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.377314 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377161    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-oauth-serving-cert\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.377314 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377198    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hc2zg\" (UniqueName: \"kubernetes.io/projected/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-kube-api-access-hc2zg\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.377314 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377225    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-config\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.377314 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377246    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-oauth-config\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.377873 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377841    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-service-ca\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.378054 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377963    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-oauth-serving-cert\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.378054 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.378049    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-trusted-ca-bundle\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.378210 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.377970    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-config\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.379114 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.379065    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:26.380283 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.380258    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-oauth-config\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.380404 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.380387    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-serving-cert\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.386190 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.386165    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hc2zg\" (UniqueName: \"kubernetes.io/projected/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-kube-api-access-hc2zg\") pod \"console-55f599b986-6ddpp\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") " pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.480971 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.480867    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:26.626950 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:26.626921    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-55f599b986-6ddpp"]
Jul 16 21:54:26.629893 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:54:26.629862    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ef7a0b3_0ede_4e02_963e_a60ac6c70fbb.slice/crio-a05e3c8152b463775995701c6718973ae562ae968da131d3cb4964d9a1bfe3ec WatchSource:0}: Error finding container a05e3c8152b463775995701c6718973ae562ae968da131d3cb4964d9a1bfe3ec: Status 404 returned error can't find the container with id a05e3c8152b463775995701c6718973ae562ae968da131d3cb4964d9a1bfe3ec
Jul 16 21:54:27.184437 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:27.184397    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:27.234722 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:27.234676    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f599b986-6ddpp" event={"ID":"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb","Type":"ContainerStarted","Data":"b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d"}
Jul 16 21:54:27.234926 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:27.234739    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f599b986-6ddpp" event={"ID":"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb","Type":"ContainerStarted","Data":"a05e3c8152b463775995701c6718973ae562ae968da131d3cb4964d9a1bfe3ec"}
Jul 16 21:54:27.257888 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:27.257823    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-55f599b986-6ddpp" podStartSLOduration=1.25780255 podStartE2EDuration="1.25780255s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:54:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:54:27.25622894 +0000 UTC m=+267.060652267" watchObservedRunningTime="2026-07-16 21:54:27.25780255 +0000 UTC m=+267.062225878"
Jul 16 21:54:27.300268 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:27.300230    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:30.456761 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:30.456722    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:30.581045 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:30.581008    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:30.668719 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:30.668682    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:36.481647 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:36.481601    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:36.481647 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:36.481664    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:36.486411 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:36.486384    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:37.283183 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:37.283139    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:54:37.345076 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:37.345038    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6477dbdbf-5qp9f"]
Jul 16 21:54:37.464466 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:37.464432    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:37.564555 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:37.564449    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:37.640598 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:37.640566    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:54:55.824627 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:54:55.824591    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:00.637185 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:00.637156    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 21:55:00.642696 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:00.642659    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 21:55:00.648390 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:00.648362    2585 kubelet.go:1648] "Image garbage collection succeeded"
Jul 16 21:55:02.367029 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.366983    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-6477dbdbf-5qp9f" podUID="481531eb-3e1e-463a-a192-8d079f97372d" containerName="console" containerID="cri-o://b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e" gracePeriod=15
Jul 16 21:55:02.612504 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.612473    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:55:02.703661 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.703558    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtc4t\" (UniqueName: \"kubernetes.io/projected/481531eb-3e1e-463a-a192-8d079f97372d-kube-api-access-wtc4t\") pod \"481531eb-3e1e-463a-a192-8d079f97372d\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") "
Jul 16 21:55:02.703661 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.703604    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-console-config\") pod \"481531eb-3e1e-463a-a192-8d079f97372d\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") "
Jul 16 21:55:02.703661 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.703649    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-serving-cert\") pod \"481531eb-3e1e-463a-a192-8d079f97372d\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") "
Jul 16 21:55:02.703941 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.703675    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-oauth-config\") pod \"481531eb-3e1e-463a-a192-8d079f97372d\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") "
Jul 16 21:55:02.703941 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.703705    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-trusted-ca-bundle\") pod \"481531eb-3e1e-463a-a192-8d079f97372d\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") "
Jul 16 21:55:02.703941 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.703724    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-oauth-serving-cert\") pod \"481531eb-3e1e-463a-a192-8d079f97372d\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") "
Jul 16 21:55:02.703941 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.703743    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-service-ca\") pod \"481531eb-3e1e-463a-a192-8d079f97372d\" (UID: \"481531eb-3e1e-463a-a192-8d079f97372d\") "
Jul 16 21:55:02.704203 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.704166    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "481531eb-3e1e-463a-a192-8d079f97372d" (UID: "481531eb-3e1e-463a-a192-8d079f97372d"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:55:02.704277 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.704170    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-console-config" (OuterVolumeSpecName: "console-config") pod "481531eb-3e1e-463a-a192-8d079f97372d" (UID: "481531eb-3e1e-463a-a192-8d079f97372d"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:55:02.704330 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.704274    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "481531eb-3e1e-463a-a192-8d079f97372d" (UID: "481531eb-3e1e-463a-a192-8d079f97372d"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:55:02.704330 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.704324    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-service-ca" (OuterVolumeSpecName: "service-ca") pod "481531eb-3e1e-463a-a192-8d079f97372d" (UID: "481531eb-3e1e-463a-a192-8d079f97372d"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:55:02.706052 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.706022    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "481531eb-3e1e-463a-a192-8d079f97372d" (UID: "481531eb-3e1e-463a-a192-8d079f97372d"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:55:02.706497 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.706466    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "481531eb-3e1e-463a-a192-8d079f97372d" (UID: "481531eb-3e1e-463a-a192-8d079f97372d"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:55:02.706543 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.706480    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/481531eb-3e1e-463a-a192-8d079f97372d-kube-api-access-wtc4t" (OuterVolumeSpecName: "kube-api-access-wtc4t") pod "481531eb-3e1e-463a-a192-8d079f97372d" (UID: "481531eb-3e1e-463a-a192-8d079f97372d"). InnerVolumeSpecName "kube-api-access-wtc4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:55:02.804958 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.804917    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:02.804958 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.804948    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/481531eb-3e1e-463a-a192-8d079f97372d-console-oauth-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:02.804958 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.804969    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-trusted-ca-bundle\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:02.805311 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.804983    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-oauth-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:02.805311 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.804993    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-service-ca\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:02.805311 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.805002    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtc4t\" (UniqueName: \"kubernetes.io/projected/481531eb-3e1e-463a-a192-8d079f97372d-kube-api-access-wtc4t\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:02.805311 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:02.805010    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/481531eb-3e1e-463a-a192-8d079f97372d-console-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:03.395068 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.395031    2585 generic.go:356] "Generic (PLEG): container finished" podID="481531eb-3e1e-463a-a192-8d079f97372d" containerID="b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e" exitCode=0
Jul 16 21:55:03.395569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.395121    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6477dbdbf-5qp9f"
Jul 16 21:55:03.395569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.395126    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477dbdbf-5qp9f" event={"ID":"481531eb-3e1e-463a-a192-8d079f97372d","Type":"ContainerDied","Data":"b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e"}
Jul 16 21:55:03.395569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.395188    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6477dbdbf-5qp9f" event={"ID":"481531eb-3e1e-463a-a192-8d079f97372d","Type":"ContainerDied","Data":"ab86803c6430bb93c9f44453a5f8cb71c695dca566c6d8fb4fa00649e21c5e60"}
Jul 16 21:55:03.395569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.395210    2585 scope.go:117] "RemoveContainer" containerID="b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e"
Jul 16 21:55:03.403869 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.403850    2585 scope.go:117] "RemoveContainer" containerID="b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e"
Jul 16 21:55:03.404166 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:03.404145    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e\": container with ID starting with b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e not found: ID does not exist" containerID="b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e"
Jul 16 21:55:03.404233 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.404175    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e"} err="failed to get container status \"b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e\": rpc error: code = NotFound desc = could not find container \"b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e\": container with ID starting with b7075c8f02776b5801665e37830e60e584b4b67b99e95d553d2ecc24afca955e not found: ID does not exist"
Jul 16 21:55:03.414307 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.414275    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6477dbdbf-5qp9f"]
Jul 16 21:55:03.418384 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:03.418357    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6477dbdbf-5qp9f"]
Jul 16 21:55:04.826153 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:04.826093    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="481531eb-3e1e-463a-a192-8d079f97372d" path="/var/lib/kubelet/pods/481531eb-3e1e-463a-a192-8d079f97372d/volumes"
Jul 16 21:55:26.228177 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:26.228094    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:26.325935 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:26.325898    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:26.818342 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:26.818304    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:26.909339 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:26.909301    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:27.009294 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.009252    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:27.848563 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.848528    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"]
Jul 16 21:55:27.848960 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.848899    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="481531eb-3e1e-463a-a192-8d079f97372d" containerName="console"
Jul 16 21:55:27.848960 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.848910    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="481531eb-3e1e-463a-a192-8d079f97372d" containerName="console"
Jul 16 21:55:27.849040 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.848987    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="481531eb-3e1e-463a-a192-8d079f97372d" containerName="console"
Jul 16 21:55:27.852371 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.852349    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:27.858706 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.858672    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\""
Jul 16 21:55:27.859430 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.859408    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\""
Jul 16 21:55:27.859519 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.859500    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"default-dockercfg-qlbnz\""
Jul 16 21:55:27.871269 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:27.871235    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"]
Jul 16 21:55:28.031743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.031700    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.031743 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.031741    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9lgnf\" (UniqueName: \"kubernetes.io/projected/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-kube-api-access-9lgnf\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.031960 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.031834    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.133359 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.133261    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.133359 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.133308    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9lgnf\" (UniqueName: \"kubernetes.io/projected/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-kube-api-access-9lgnf\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.133548 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.133378    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.133678 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.133656    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-bundle\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.133744 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.133723    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-util\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.143265 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.143228    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9lgnf\" (UniqueName: \"kubernetes.io/projected/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-kube-api-access-9lgnf\") pod \"320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") " pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.162242 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.162205    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:28.299993 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.299962    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"]
Jul 16 21:55:28.302833 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:55:28.302800    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod421a723c_c250_4e13_bc78_ab0c0fb8d1cc.slice/crio-38d9c5de17491b223475c7f8b88af9474ccea75de8533680c4255a6df977b59f WatchSource:0}: Error finding container 38d9c5de17491b223475c7f8b88af9474ccea75de8533680c4255a6df977b59f: Status 404 returned error can't find the container with id 38d9c5de17491b223475c7f8b88af9474ccea75de8533680c4255a6df977b59f
Jul 16 21:55:28.304792 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.304771    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 21:55:28.515155 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:28.515110    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn" event={"ID":"421a723c-c250-4e13-bc78-ab0c0fb8d1cc","Type":"ContainerStarted","Data":"38d9c5de17491b223475c7f8b88af9474ccea75de8533680c4255a6df977b59f"}
Jul 16 21:55:33.543547 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:33.543501    2585 generic.go:356] "Generic (PLEG): container finished" podID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerID="a86a407de7eaa37dd07e63e9808bd5837874a566e305d5c9dffd5ccd8894063e" exitCode=0
Jul 16 21:55:33.544003 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:33.543583    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn" event={"ID":"421a723c-c250-4e13-bc78-ab0c0fb8d1cc","Type":"ContainerDied","Data":"a86a407de7eaa37dd07e63e9808bd5837874a566e305d5c9dffd5ccd8894063e"}
Jul 16 21:55:36.561021 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:36.560981    2585 generic.go:356] "Generic (PLEG): container finished" podID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerID="ccb4d0d8e0d09403189307e627de9f0386a7e412a7aad734923086dbcc35d729" exitCode=0
Jul 16 21:55:36.561433 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:36.561066    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn" event={"ID":"421a723c-c250-4e13-bc78-ab0c0fb8d1cc","Type":"ContainerDied","Data":"ccb4d0d8e0d09403189307e627de9f0386a7e412a7aad734923086dbcc35d729"}
Jul 16 21:55:43.603795 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:43.603752    2585 generic.go:356] "Generic (PLEG): container finished" podID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerID="94b998e83362f71c3d9d8752d07ad56c64e60b19a00f1705173ae49c1947bb0a" exitCode=0
Jul 16 21:55:43.604421 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:43.603851    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn" event={"ID":"421a723c-c250-4e13-bc78-ab0c0fb8d1cc","Type":"ContainerDied","Data":"94b998e83362f71c3d9d8752d07ad56c64e60b19a00f1705173ae49c1947bb0a"}
Jul 16 21:55:44.741252 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.741222    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:44.798421 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.798384    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-util\") pod \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") "
Jul 16 21:55:44.798627 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.798442    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9lgnf\" (UniqueName: \"kubernetes.io/projected/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-kube-api-access-9lgnf\") pod \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") "
Jul 16 21:55:44.798627 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.798474    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-bundle\") pod \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\" (UID: \"421a723c-c250-4e13-bc78-ab0c0fb8d1cc\") "
Jul 16 21:55:44.799093 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.799067    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-bundle" (OuterVolumeSpecName: "bundle") pod "421a723c-c250-4e13-bc78-ab0c0fb8d1cc" (UID: "421a723c-c250-4e13-bc78-ab0c0fb8d1cc"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:44.800752 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.800725    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-kube-api-access-9lgnf" (OuterVolumeSpecName: "kube-api-access-9lgnf") pod "421a723c-c250-4e13-bc78-ab0c0fb8d1cc" (UID: "421a723c-c250-4e13-bc78-ab0c0fb8d1cc"). InnerVolumeSpecName "kube-api-access-9lgnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:55:44.803248 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.803223    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-util" (OuterVolumeSpecName: "util") pod "421a723c-c250-4e13-bc78-ab0c0fb8d1cc" (UID: "421a723c-c250-4e13-bc78-ab0c0fb8d1cc"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 16 21:55:44.900036 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.899930    2585 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-util\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:44.900036 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.899970    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9lgnf\" (UniqueName: \"kubernetes.io/projected/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-kube-api-access-9lgnf\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:44.900036 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:44.899985    2585 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/421a723c-c250-4e13-bc78-ab0c0fb8d1cc-bundle\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:55:45.614063 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:45.614033    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn"
Jul 16 21:55:45.614251 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:45.614065    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/320da698f62953fa8a11a45982d6c7ee4dcf44b832e3de43010dcb42b5hj5wn" event={"ID":"421a723c-c250-4e13-bc78-ab0c0fb8d1cc","Type":"ContainerDied","Data":"38d9c5de17491b223475c7f8b88af9474ccea75de8533680c4255a6df977b59f"}
Jul 16 21:55:45.614251 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:45.614131    2585 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="38d9c5de17491b223475c7f8b88af9474ccea75de8533680c4255a6df977b59f"
Jul 16 21:55:49.349903 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:49.349864    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:51.069064 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.069028    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"]
Jul 16 21:55:51.069453 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.069409    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerName="pull"
Jul 16 21:55:51.069453 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.069421    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerName="pull"
Jul 16 21:55:51.069453 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.069429    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerName="util"
Jul 16 21:55:51.069453 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.069434    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerName="util"
Jul 16 21:55:51.069453 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.069444    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerName="extract"
Jul 16 21:55:51.069453 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.069449    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerName="extract"
Jul 16 21:55:51.069660 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.069510    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="421a723c-c250-4e13-bc78-ab0c0fb8d1cc" containerName="extract"
Jul 16 21:55:51.073025 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.073003    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:51.075651 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.075622    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"kedaorg-certs\""
Jul 16 21:55:51.075799 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.075626    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"openshift-service-ca.crt\""
Jul 16 21:55:51.075866 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.075834    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"custom-metrics-autoscaler-operator-dockercfg-rr6cg\""
Jul 16 21:55:51.075925 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.075893    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"kube-root-ca.crt\""
Jul 16 21:55:51.087452 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.087421    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"]
Jul 16 21:55:51.133398 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.133330    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:51.155521 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.155486    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5rh5\" (UniqueName: \"kubernetes.io/projected/cad75d40-4d91-428b-91f6-2534e9ecaefb-kube-api-access-z5rh5\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-rqz29\" (UID: \"cad75d40-4d91-428b-91f6-2534e9ecaefb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:51.155665 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.155630    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/cad75d40-4d91-428b-91f6-2534e9ecaefb-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-rqz29\" (UID: \"cad75d40-4d91-428b-91f6-2534e9ecaefb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:51.256548 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.256493    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5rh5\" (UniqueName: \"kubernetes.io/projected/cad75d40-4d91-428b-91f6-2534e9ecaefb-kube-api-access-z5rh5\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-rqz29\" (UID: \"cad75d40-4d91-428b-91f6-2534e9ecaefb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:51.256748 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.256605    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/cad75d40-4d91-428b-91f6-2534e9ecaefb-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-rqz29\" (UID: \"cad75d40-4d91-428b-91f6-2534e9ecaefb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:51.259066 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.259041    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/secret/cad75d40-4d91-428b-91f6-2534e9ecaefb-certificates\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-rqz29\" (UID: \"cad75d40-4d91-428b-91f6-2534e9ecaefb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:51.266569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.266540    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5rh5\" (UniqueName: \"kubernetes.io/projected/cad75d40-4d91-428b-91f6-2534e9ecaefb-kube-api-access-z5rh5\") pod \"custom-metrics-autoscaler-operator-6969cb7dfb-rqz29\" (UID: \"cad75d40-4d91-428b-91f6-2534e9ecaefb\") " pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:51.384068 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.383978    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:51.665451 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:51.665418    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"]
Jul 16 21:55:51.667436 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:55:51.667403    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcad75d40_4d91_428b_91f6_2534e9ecaefb.slice/crio-1e1236ec6d36e9c9eae01ae9128d23af6f8a883b1bc3928a0def427e64f1dab1 WatchSource:0}: Error finding container 1e1236ec6d36e9c9eae01ae9128d23af6f8a883b1bc3928a0def427e64f1dab1: Status 404 returned error can't find the container with id 1e1236ec6d36e9c9eae01ae9128d23af6f8a883b1bc3928a0def427e64f1dab1
Jul 16 21:55:52.649897 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:52.649810    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29" event={"ID":"cad75d40-4d91-428b-91f6-2534e9ecaefb","Type":"ContainerStarted","Data":"1e1236ec6d36e9c9eae01ae9128d23af6f8a883b1bc3928a0def427e64f1dab1"}
Jul 16 21:55:54.648196 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.648145    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:54.662068 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.662033    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29" event={"ID":"cad75d40-4d91-428b-91f6-2534e9ecaefb","Type":"ContainerStarted","Data":"19fa5620f19bc3ae884f6a1eda161e3da20a171731fb8653b6838a93f8b0fb8f"}
Jul 16 21:55:54.662270 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.662157    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:55:54.688079 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.688011    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29" podStartSLOduration=1.160916523 podStartE2EDuration="3.687993112s" totalImagesPullingTime="2.527076589s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:55:54.686507064 +0000 UTC m=+354.490930389" watchObservedRunningTime="2026-07-16 21:55:54.687993112 +0000 UTC m=+354.492416437"
Jul 16 21:55:54.983704 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.983662    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-operator-77f648d694-wfmpm"]
Jul 16 21:55:54.985456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.985428    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:54.987562 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.987548    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:54.994259 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.994230    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-certs\""
Jul 16 21:55:54.994410 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.994278    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-operator-dockercfg-nms79\""
Jul 16 21:55:54.994685 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:54.994666    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-keda\"/\"keda-ocp-cabundle\""
Jul 16 21:55:55.002698 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.002669    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-wfmpm"]
Jul 16 21:55:55.094627 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.094584    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brx8s\" (UniqueName: \"kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-kube-api-access-brx8s\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.094808 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.094648    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/31f58887-6291-47d3-b791-8f640037fb9d-cabundle0\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.094808 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.094715    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-certificates\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.195613 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.195572    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brx8s\" (UniqueName: \"kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-kube-api-access-brx8s\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.195613 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.195620    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/31f58887-6291-47d3-b791-8f640037fb9d-cabundle0\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.195862 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.195656    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-certificates\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.195862 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.195767    2585 secret.go:281] references non-existent secret key: ca.crt
Jul 16 21:55:55.195862 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.195781    2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: ca.crt
Jul 16 21:55:55.195862 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.195790    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-operator-77f648d694-wfmpm: references non-existent secret key: ca.crt
Jul 16 21:55:55.195862 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.195842    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-certificates podName:31f58887-6291-47d3-b791-8f640037fb9d nodeName:}" failed. No retries permitted until 2026-07-16 21:55:55.695825349 +0000 UTC m=+355.500248652 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-certificates") pod "keda-operator-77f648d694-wfmpm" (UID: "31f58887-6291-47d3-b791-8f640037fb9d") : references non-existent secret key: ca.crt
Jul 16 21:55:55.196305 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.196286    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cabundle0\" (UniqueName: \"kubernetes.io/configmap/31f58887-6291-47d3-b791-8f640037fb9d-cabundle0\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.210183 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.210139    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-brx8s\" (UniqueName: \"kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-kube-api-access-brx8s\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.216484 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.216450    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:55.217884 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.217854    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"]
Jul 16 21:55:55.221543 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.221523    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.224882 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.224858    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-metrics-apiserver-certs\""
Jul 16 21:55:55.237259 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.237190    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"]
Jul 16 21:55:55.295517 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.295477    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:55.296442 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.296422    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-certificates\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.296505 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.296481    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1cfee299-767f-419e-b654-8b83f1f1f1ac-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.296557 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.296517    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvjkp\" (UniqueName: \"kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-kube-api-access-fvjkp\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.397800 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.397755    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-certificates\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.397977 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.397850    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1cfee299-767f-419e-b654-8b83f1f1f1ac-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.397977 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.397898    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvjkp\" (UniqueName: \"kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-kube-api-access-fvjkp\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.397977 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.397919    2585 secret.go:281] references non-existent secret key: tls.crt
Jul 16 21:55:55.397977 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.397939    2585 projected.go:279] Couldn't get secret payload openshift-keda/kedaorg-certs: references non-existent secret key: tls.crt
Jul 16 21:55:55.397977 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.397961    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx: references non-existent secret key: tls.crt
Jul 16 21:55:55.398180 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.398024    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-certificates podName:1cfee299-767f-419e-b654-8b83f1f1f1ac nodeName:}" failed. No retries permitted until 2026-07-16 21:55:55.89800666 +0000 UTC m=+355.702429978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-certificates") pod "keda-metrics-apiserver-55557df8c-tp5wx" (UID: "1cfee299-767f-419e-b654-8b83f1f1f1ac") : references non-existent secret key: tls.crt
Jul 16 21:55:55.398305 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.398282    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"temp-vol\" (UniqueName: \"kubernetes.io/empty-dir/1cfee299-767f-419e-b654-8b83f1f1f1ac-temp-vol\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.407114 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.407077    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvjkp\" (UniqueName: \"kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-kube-api-access-fvjkp\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.461573 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.461533    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:55.686759 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.686665    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:55.699865 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.699829    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-6rts5"]
Jul 16 21:55:55.700234 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.700214    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-certificates\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.702815 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.702789    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/31f58887-6291-47d3-b791-8f640037fb9d-certificates\") pod \"keda-operator-77f648d694-wfmpm\" (UID: \"31f58887-6291-47d3-b791-8f640037fb9d\") " pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.703359 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.703343    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:55.705700 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.705677    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-keda\"/\"keda-admission-webhooks-certs\""
Jul 16 21:55:55.714214 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.714184    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-6rts5"]
Jul 16 21:55:55.786695 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.786650    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:55.801584 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.800780    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c9ds\" (UniqueName: \"kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-kube-api-access-9c9ds\") pod \"keda-admission-56f9798bb4-6rts5\" (UID: \"a632bdee-c2f9-4b0d-84f8-a620f3d9632d\") " pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:55.801921 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.801902    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-certificates\") pod \"keda-admission-56f9798bb4-6rts5\" (UID: \"a632bdee-c2f9-4b0d-84f8-a620f3d9632d\") " pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:55.891488 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.891448    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:55.898251 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.898216    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:55:55.903662 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.903595    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-certificates\") pod \"keda-admission-56f9798bb4-6rts5\" (UID: \"a632bdee-c2f9-4b0d-84f8-a620f3d9632d\") " pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:55.903896 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.903728    2585 projected.go:266] Couldn't get secret openshift-keda/keda-admission-webhooks-certs: secret "keda-admission-webhooks-certs" not found
Jul 16 21:55:55.903971 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.903913    2585 projected.go:196] Error preparing data for projected volume certificates for pod openshift-keda/keda-admission-56f9798bb4-6rts5: secret "keda-admission-webhooks-certs" not found
Jul 16 21:55:55.904030 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:55:55.903989    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-certificates podName:a632bdee-c2f9-4b0d-84f8-a620f3d9632d nodeName:}" failed. No retries permitted until 2026-07-16 21:55:56.403965345 +0000 UTC m=+356.208388662 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "certificates" (UniqueName: "kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-certificates") pod "keda-admission-56f9798bb4-6rts5" (UID: "a632bdee-c2f9-4b0d-84f8-a620f3d9632d") : secret "keda-admission-webhooks-certs" not found
Jul 16 21:55:55.904255 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.904221    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c9ds\" (UniqueName: \"kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-kube-api-access-9c9ds\") pod \"keda-admission-56f9798bb4-6rts5\" (UID: \"a632bdee-c2f9-4b0d-84f8-a620f3d9632d\") " pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:55.904406 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.904382    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-certificates\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.907145 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.907118    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/1cfee299-767f-419e-b654-8b83f1f1f1ac-certificates\") pod \"keda-metrics-apiserver-55557df8c-tp5wx\" (UID: \"1cfee299-767f-419e-b654-8b83f1f1f1ac\") " pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:55.913281 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:55.913249    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c9ds\" (UniqueName: \"kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-kube-api-access-9c9ds\") pod \"keda-admission-56f9798bb4-6rts5\" (UID: \"a632bdee-c2f9-4b0d-84f8-a620f3d9632d\") " pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:56.071298 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.071269    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-operator-77f648d694-wfmpm"]
Jul 16 21:55:56.075318 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:55:56.075276    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f58887_6291_47d3_b791_8f640037fb9d.slice/crio-9b0b4f9386e6475eaa5c1b3268e17a2072b91d78e6908a847b056ca1338346fe WatchSource:0}: Error finding container 9b0b4f9386e6475eaa5c1b3268e17a2072b91d78e6908a847b056ca1338346fe: Status 404 returned error can't find the container with id 9b0b4f9386e6475eaa5c1b3268e17a2072b91d78e6908a847b056ca1338346fe
Jul 16 21:55:56.137622 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.137580    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:55:56.285018 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.284981    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:56.333433 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.333407    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"]
Jul 16 21:55:56.336866 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:55:56.336831    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cfee299_767f_419e_b654_8b83f1f1f1ac.slice/crio-e32332fa399ad47da0960b2a7741ab64fbdc18082b7680958608aac75e388b4a WatchSource:0}: Error finding container e32332fa399ad47da0960b2a7741ab64fbdc18082b7680958608aac75e388b4a: Status 404 returned error can't find the container with id e32332fa399ad47da0960b2a7741ab64fbdc18082b7680958608aac75e388b4a
Jul 16 21:55:56.395306 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.395259    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:56.412533 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.412485    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-certificates\") pod \"keda-admission-56f9798bb4-6rts5\" (UID: \"a632bdee-c2f9-4b0d-84f8-a620f3d9632d\") " pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:56.416156 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.416127    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"certificates\" (UniqueName: \"kubernetes.io/projected/a632bdee-c2f9-4b0d-84f8-a620f3d9632d-certificates\") pod \"keda-admission-56f9798bb4-6rts5\" (UID: \"a632bdee-c2f9-4b0d-84f8-a620f3d9632d\") " pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:56.598785 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.598683    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:56.622655 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.622623    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:55:56.673515 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.673479    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx" event={"ID":"1cfee299-767f-419e-b654-8b83f1f1f1ac","Type":"ContainerStarted","Data":"e32332fa399ad47da0960b2a7741ab64fbdc18082b7680958608aac75e388b4a"}
Jul 16 21:55:56.674721 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.674696    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-wfmpm" event={"ID":"31f58887-6291-47d3-b791-8f640037fb9d","Type":"ContainerStarted","Data":"9b0b4f9386e6475eaa5c1b3268e17a2072b91d78e6908a847b056ca1338346fe"}
Jul 16 21:55:56.698746 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.698707    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:55:56.787379 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:56.787306    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-keda/keda-admission-56f9798bb4-6rts5"]
Jul 16 21:55:56.790143 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:55:56.790109    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda632bdee_c2f9_4b0d_84f8_a620f3d9632d.slice/crio-0dbb949833734ffdf89f724cf247a651284808128e8456524ec332535c6c5c98 WatchSource:0}: Error finding container 0dbb949833734ffdf89f724cf247a651284808128e8456524ec332535c6c5c98: Status 404 returned error can't find the container with id 0dbb949833734ffdf89f724cf247a651284808128e8456524ec332535c6c5c98
Jul 16 21:55:57.682737 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:55:57.682684    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-6rts5" event={"ID":"a632bdee-c2f9-4b0d-84f8-a620f3d9632d","Type":"ContainerStarted","Data":"0dbb949833734ffdf89f724cf247a651284808128e8456524ec332535c6c5c98"}
Jul 16 21:56:00.701780 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:00.701737    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx" event={"ID":"1cfee299-767f-419e-b654-8b83f1f1f1ac","Type":"ContainerStarted","Data":"230457c47e2ff45081610a46597a6e0aed7976150500ff73cca9336d4f09cf10"}
Jul 16 21:56:00.702276 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:00.701845    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:56:00.703432 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:00.703406    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-operator-77f648d694-wfmpm" event={"ID":"31f58887-6291-47d3-b791-8f640037fb9d","Type":"ContainerStarted","Data":"4953f9ff2762f5a7ae9b347ce0c2753eaf2d44e85ac108a9bae5a9de23b0a3dc"}
Jul 16 21:56:00.703572 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:00.703458    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:56:00.725941 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:00.725872    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx" podStartSLOduration=1.984650507 podStartE2EDuration="5.725854733s" totalImagesPullingTime="3.741204226s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:00.723498705 +0000 UTC m=+360.527922030" watchObservedRunningTime="2026-07-16 21:56:00.725854733 +0000 UTC m=+360.530278126"
Jul 16 21:56:06.733739 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:06.733700    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-keda/keda-admission-56f9798bb4-6rts5" event={"ID":"a632bdee-c2f9-4b0d-84f8-a620f3d9632d","Type":"ContainerStarted","Data":"f9d6de665fd3b4aebbe48b3184cfbd2adc29d8ac841fbcc2dbfc1176a676916e"}
Jul 16 21:56:06.734200 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:06.733766    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:56:06.754455 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:06.754397    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-operator-77f648d694-wfmpm" podStartSLOduration=8.745106912 podStartE2EDuration="12.754379263s" totalImagesPullingTime="4.009272351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:00.750677367 +0000 UTC m=+360.555100692" watchObservedRunningTime="2026-07-16 21:56:06.754379263 +0000 UTC m=+366.558802588"
Jul 16 21:56:06.754626 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:06.754499    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-keda/keda-admission-56f9798bb4-6rts5" podStartSLOduration=2.135149905 podStartE2EDuration="11.754494543s" totalImagesPullingTime="9.619344638s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:55:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:56:06.753183942 +0000 UTC m=+366.557607267" watchObservedRunningTime="2026-07-16 21:56:06.754494543 +0000 UTC m=+366.558917870"
Jul 16 21:56:11.713259 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:11.713227    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-metrics-apiserver-55557df8c-tp5wx"
Jul 16 21:56:15.670067 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:15.670028    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/custom-metrics-autoscaler-operator-6969cb7dfb-rqz29"
Jul 16 21:56:21.710982 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:21.710947    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-operator-77f648d694-wfmpm"
Jul 16 21:56:27.740799 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:56:27.740755    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-keda/keda-admission-56f9798bb4-6rts5"
Jul 16 21:57:13.441248 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.441206    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:13.557880 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.557843    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-hl569"]
Jul 16 21:57:13.561583 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.561557    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:13.564255 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.564230    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-webhook-server-cert\""
Jul 16 21:57:13.564598 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.564572    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"openshift-service-ca.crt\""
Jul 16 21:57:13.565222 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.565206    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve\"/\"kube-root-ca.crt\""
Jul 16 21:57:13.565280 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.565214    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"kserve-controller-manager-dockercfg-9dnth\""
Jul 16 21:57:13.576850 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.576812    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:13.585909 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.585877    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-hl569"]
Jul 16 21:57:13.609350 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.609316    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2dvn\" (UniqueName: \"kubernetes.io/projected/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-kube-api-access-h2dvn\") pod \"kserve-controller-manager-84cf6d888d-hl569\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") " pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:13.609527 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.609402    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert\") pod \"kserve-controller-manager-84cf6d888d-hl569\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") " pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:13.626803 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.626761    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/seaweedfs-64568bcd49-fskld"]
Jul 16 21:57:13.636707 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.636675    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:13.640343 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.640296    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"default-dockercfg-8pwwf\""
Jul 16 21:57:13.640343 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.640321    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve\"/\"mlpipeline-s3-artifact\""
Jul 16 21:57:13.648390 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.648361    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-fskld"]
Jul 16 21:57:13.708736 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.708644    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:13.710738 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.710711    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2dvn\" (UniqueName: \"kubernetes.io/projected/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-kube-api-access-h2dvn\") pod \"kserve-controller-manager-84cf6d888d-hl569\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") " pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:13.710828 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.710803    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert\") pod \"kserve-controller-manager-84cf6d888d-hl569\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") " pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:13.710866 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.710839    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bed0b130-8354-4c42-82be-a0190f19c752-data\") pod \"seaweedfs-64568bcd49-fskld\" (UID: \"bed0b130-8354-4c42-82be-a0190f19c752\") " pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:13.710912 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.710883    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r697g\" (UniqueName: \"kubernetes.io/projected/bed0b130-8354-4c42-82be-a0190f19c752-kube-api-access-r697g\") pod \"seaweedfs-64568bcd49-fskld\" (UID: \"bed0b130-8354-4c42-82be-a0190f19c752\") " pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:13.710967 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:57:13.710951    2585 secret.go:189] Couldn't get secret kserve/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 16 21:57:13.711052 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:57:13.711042    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert podName:ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08 nodeName:}" failed. No retries permitted until 2026-07-16 21:57:14.211025844 +0000 UTC m=+434.015449162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert") pod "kserve-controller-manager-84cf6d888d-hl569" (UID: "ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08") : secret "kserve-webhook-server-cert" not found
Jul 16 21:57:13.724554 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.724516    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2dvn\" (UniqueName: \"kubernetes.io/projected/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-kube-api-access-h2dvn\") pod \"kserve-controller-manager-84cf6d888d-hl569\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") " pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:13.812050 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.812006    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r697g\" (UniqueName: \"kubernetes.io/projected/bed0b130-8354-4c42-82be-a0190f19c752-kube-api-access-r697g\") pod \"seaweedfs-64568bcd49-fskld\" (UID: \"bed0b130-8354-4c42-82be-a0190f19c752\") " pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:13.812261 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.812183    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bed0b130-8354-4c42-82be-a0190f19c752-data\") pod \"seaweedfs-64568bcd49-fskld\" (UID: \"bed0b130-8354-4c42-82be-a0190f19c752\") " pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:13.812591 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.812572    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/bed0b130-8354-4c42-82be-a0190f19c752-data\") pod \"seaweedfs-64568bcd49-fskld\" (UID: \"bed0b130-8354-4c42-82be-a0190f19c752\") " pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:13.823872 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.823839    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r697g\" (UniqueName: \"kubernetes.io/projected/bed0b130-8354-4c42-82be-a0190f19c752-kube-api-access-r697g\") pod \"seaweedfs-64568bcd49-fskld\" (UID: \"bed0b130-8354-4c42-82be-a0190f19c752\") " pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:13.835895 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.835864    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:13.936397 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.936364    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:13.947306 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:13.947264    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:14.089814 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:14.089700    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/seaweedfs-64568bcd49-fskld"]
Jul 16 21:57:14.092676 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:57:14.092646    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbed0b130_8354_4c42_82be_a0190f19c752.slice/crio-3d80310f350123e8d1962772e009c7b816d561823e7b88ae45561dc48332099a WatchSource:0}: Error finding container 3d80310f350123e8d1962772e009c7b816d561823e7b88ae45561dc48332099a: Status 404 returned error can't find the container with id 3d80310f350123e8d1962772e009c7b816d561823e7b88ae45561dc48332099a
Jul 16 21:57:14.216692 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:14.216591    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert\") pod \"kserve-controller-manager-84cf6d888d-hl569\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") " pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:14.219215 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:14.219178    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert\") pod \"kserve-controller-manager-84cf6d888d-hl569\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") " pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:14.242468 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:14.242429    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:14.337448 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:14.337416    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:57:14.472952 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:14.472857    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:14.637369 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:14.637329    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-hl569"]
Jul 16 21:57:14.641592 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:57:14.641552    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac5e1fa0_c8b6_4f9d_b067_8c75ea5fcf08.slice/crio-24c4936f53ae636bd6c6f077a8454413babdf674ea00d68843c21d7b9c6ce2f7 WatchSource:0}: Error finding container 24c4936f53ae636bd6c6f077a8454413babdf674ea00d68843c21d7b9c6ce2f7: Status 404 returned error can't find the container with id 24c4936f53ae636bd6c6f077a8454413babdf674ea00d68843c21d7b9c6ce2f7
Jul 16 21:57:15.063181 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:15.063136    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-fskld" event={"ID":"bed0b130-8354-4c42-82be-a0190f19c752","Type":"ContainerStarted","Data":"3d80310f350123e8d1962772e009c7b816d561823e7b88ae45561dc48332099a"}
Jul 16 21:57:15.064659 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:15.064632    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-hl569" event={"ID":"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08","Type":"ContainerStarted","Data":"24c4936f53ae636bd6c6f077a8454413babdf674ea00d68843c21d7b9c6ce2f7"}
Jul 16 21:57:19.094806 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:19.094763    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-hl569" event={"ID":"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08","Type":"ContainerStarted","Data":"5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1"}
Jul 16 21:57:19.095291 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:19.094859    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:19.096453 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:19.096426    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/seaweedfs-64568bcd49-fskld" event={"ID":"bed0b130-8354-4c42-82be-a0190f19c752","Type":"ContainerStarted","Data":"c3ca5b60cbbcbf97f2102186f69d967f9e1e66bd29b26e43767cf69a00f2e13d"}
Jul 16 21:57:19.096582 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:19.096567    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:19.115396 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:19.115343    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-84cf6d888d-hl569" podStartSLOduration=2.698544441 podStartE2EDuration="6.115327746s" totalImagesPullingTime="3.416783305s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:19.114536451 +0000 UTC m=+438.918959778" watchObservedRunningTime="2026-07-16 21:57:19.115327746 +0000 UTC m=+438.919751071"
Jul 16 21:57:19.132976 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:19.132918    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/seaweedfs-64568bcd49-fskld" podStartSLOduration=2.114575437 podStartE2EDuration="6.132899328s" totalImagesPullingTime="4.018323891s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:19.13195122 +0000 UTC m=+438.936374547" watchObservedRunningTime="2026-07-16 21:57:19.132899328 +0000 UTC m=+438.937322653"
Jul 16 21:57:25.104598 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:25.104511    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/seaweedfs-64568bcd49-fskld"
Jul 16 21:57:50.107631 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:50.107588    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:56.731476 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.731431    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-hl569"]
Jul 16 21:57:56.731947 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.731682    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve/kserve-controller-manager-84cf6d888d-hl569" podUID="ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08" containerName="manager" containerID="cri-o://5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1" gracePeriod=10
Jul 16 21:57:56.756455 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.756422    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-gttm2"]
Jul 16 21:57:56.759946 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.759920    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:56.770861 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.770823    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-gttm2"]
Jul 16 21:57:56.811962 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.811922    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a742894-0fa6-49eb-bbb3-f5be3cedc88f-cert\") pod \"kserve-controller-manager-84cf6d888d-gttm2\" (UID: \"6a742894-0fa6-49eb-bbb3-f5be3cedc88f\") " pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:56.812145 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.812031    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8fnw\" (UniqueName: \"kubernetes.io/projected/6a742894-0fa6-49eb-bbb3-f5be3cedc88f-kube-api-access-b8fnw\") pod \"kserve-controller-manager-84cf6d888d-gttm2\" (UID: \"6a742894-0fa6-49eb-bbb3-f5be3cedc88f\") " pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:56.913222 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.912986    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a742894-0fa6-49eb-bbb3-f5be3cedc88f-cert\") pod \"kserve-controller-manager-84cf6d888d-gttm2\" (UID: \"6a742894-0fa6-49eb-bbb3-f5be3cedc88f\") " pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:56.913645 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.913614    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8fnw\" (UniqueName: \"kubernetes.io/projected/6a742894-0fa6-49eb-bbb3-f5be3cedc88f-kube-api-access-b8fnw\") pod \"kserve-controller-manager-84cf6d888d-gttm2\" (UID: \"6a742894-0fa6-49eb-bbb3-f5be3cedc88f\") " pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:56.915865 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.915830    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6a742894-0fa6-49eb-bbb3-f5be3cedc88f-cert\") pod \"kserve-controller-manager-84cf6d888d-gttm2\" (UID: \"6a742894-0fa6-49eb-bbb3-f5be3cedc88f\") " pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:56.923239 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.923206    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8fnw\" (UniqueName: \"kubernetes.io/projected/6a742894-0fa6-49eb-bbb3-f5be3cedc88f-kube-api-access-b8fnw\") pod \"kserve-controller-manager-84cf6d888d-gttm2\" (UID: \"6a742894-0fa6-49eb-bbb3-f5be3cedc88f\") " pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:56.981899 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:56.981833    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:57.014148 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.014114    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert\") pod \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") "
Jul 16 21:57:57.014311 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.014188    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h2dvn\" (UniqueName: \"kubernetes.io/projected/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-kube-api-access-h2dvn\") pod \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\" (UID: \"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08\") "
Jul 16 21:57:57.016449 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.016412    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-kube-api-access-h2dvn" (OuterVolumeSpecName: "kube-api-access-h2dvn") pod "ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08" (UID: "ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08"). InnerVolumeSpecName "kube-api-access-h2dvn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:57:57.016601 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.016469    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert" (OuterVolumeSpecName: "cert") pod "ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08" (UID: "ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08"). InnerVolumeSpecName "cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:57:57.109967 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.109926    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:57.115547 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.115521    2585 reconciler_common.go:299] "Volume detached for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:57:57.115608 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.115557    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2dvn\" (UniqueName: \"kubernetes.io/projected/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08-kube-api-access-h2dvn\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:57:57.257721 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.257629    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-gttm2"]
Jul 16 21:57:57.261171 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:57:57.261131    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a742894_0fa6_49eb_bbb3_f5be3cedc88f.slice/crio-4336443b9827ee193ee4d5f07116e870b1debaf5f4dc3e53d40a1405aba570a5 WatchSource:0}: Error finding container 4336443b9827ee193ee4d5f07116e870b1debaf5f4dc3e53d40a1405aba570a5: Status 404 returned error can't find the container with id 4336443b9827ee193ee4d5f07116e870b1debaf5f4dc3e53d40a1405aba570a5
Jul 16 21:57:57.281267 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.281231    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-gttm2" event={"ID":"6a742894-0fa6-49eb-bbb3-f5be3cedc88f","Type":"ContainerStarted","Data":"4336443b9827ee193ee4d5f07116e870b1debaf5f4dc3e53d40a1405aba570a5"}
Jul 16 21:57:57.282640 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.282604    2585 generic.go:356] "Generic (PLEG): container finished" podID="ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08" containerID="5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1" exitCode=0
Jul 16 21:57:57.282640 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.282646    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-hl569" event={"ID":"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08","Type":"ContainerDied","Data":"5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1"}
Jul 16 21:57:57.282848 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.282652    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve/kserve-controller-manager-84cf6d888d-hl569"
Jul 16 21:57:57.282848 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.282691    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-hl569" event={"ID":"ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08","Type":"ContainerDied","Data":"24c4936f53ae636bd6c6f077a8454413babdf674ea00d68843c21d7b9c6ce2f7"}
Jul 16 21:57:57.282848 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.282716    2585 scope.go:117] "RemoveContainer" containerID="5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1"
Jul 16 21:57:57.292038 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.292016    2585 scope.go:117] "RemoveContainer" containerID="5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1"
Jul 16 21:57:57.292462 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:57:57.292442    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1\": container with ID starting with 5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1 not found: ID does not exist" containerID="5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1"
Jul 16 21:57:57.292561 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.292476    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1"} err="failed to get container status \"5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1\": rpc error: code = NotFound desc = could not find container \"5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1\": container with ID starting with 5d9e1dfeeb3c61226e13fc7aebf5cb58fef9f72a58f2974796ea8190dba5c6e1 not found: ID does not exist"
Jul 16 21:57:57.306681 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.306648    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-hl569"]
Jul 16 21:57:57.311046 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:57.311013    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve/kserve-controller-manager-84cf6d888d-hl569"]
Jul 16 21:57:58.289718 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:58.289672    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve/kserve-controller-manager-84cf6d888d-gttm2" event={"ID":"6a742894-0fa6-49eb-bbb3-f5be3cedc88f","Type":"ContainerStarted","Data":"a32f28a279d0a579f6d9744976c30041bb7ceb7a19a765718d04d2d547abfdfc"}
Jul 16 21:57:58.290238 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:58.289745    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:57:58.310196 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:58.310138    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve/kserve-controller-manager-84cf6d888d-gttm2" podStartSLOduration=1.982336482 podStartE2EDuration="2.310120527s" totalImagesPullingTime="327.784045ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:57:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:57:58.30819743 +0000 UTC m=+478.112620756" watchObservedRunningTime="2026-07-16 21:57:58.310120527 +0000 UTC m=+478.114543847"
Jul 16 21:57:58.827013 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:57:58.826968    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08" path="/var/lib/kubelet/pods/ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08/volumes"
Jul 16 21:58:12.699797 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.699752    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-7c479d9b44-bctf4"]
Jul 16 21:58:12.700434 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.700413    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08" containerName="manager"
Jul 16 21:58:12.700483 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.700438    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08" containerName="manager"
Jul 16 21:58:12.700573 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.700562    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac5e1fa0-c8b6-4f9d-b067-8c75ea5fcf08" containerName="manager"
Jul 16 21:58:12.705703 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.705674    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.718905 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.718866    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c479d9b44-bctf4"]
Jul 16 21:58:12.756646 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.756607    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-serving-cert\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.756832 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.756658    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-trusted-ca-bundle\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.756832 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.756746    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-oauth-serving-cert\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.756832 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.756783    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-config\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.756832 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.756819    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cz7hd\" (UniqueName: \"kubernetes.io/projected/db081b6b-faa0-4113-ac3e-802dccb6d79d-kube-api-access-cz7hd\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.756959 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.756859    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-service-ca\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.756959 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.756881    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-oauth-config\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.772542 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.772502    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:12.841571 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.841540    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:12.857739 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.857700    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-serving-cert\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.857739 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.857745    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-trusted-ca-bundle\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.857947 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.857776    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-oauth-serving-cert\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.857947 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.857796    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-config\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.857947 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.857818    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cz7hd\" (UniqueName: \"kubernetes.io/projected/db081b6b-faa0-4113-ac3e-802dccb6d79d-kube-api-access-cz7hd\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.857947 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.857857    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-service-ca\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.857947 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.857884    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-oauth-config\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.858747 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.858714    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-oauth-serving-cert\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.858910 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.858730    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-service-ca\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.858910 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.858818    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-config\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.859041 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.858954    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/db081b6b-faa0-4113-ac3e-802dccb6d79d-trusted-ca-bundle\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.860404 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.860383    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-oauth-config\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.860513 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.860494    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/db081b6b-faa0-4113-ac3e-802dccb6d79d-console-serving-cert\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:12.866957 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:12.866932    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cz7hd\" (UniqueName: \"kubernetes.io/projected/db081b6b-faa0-4113-ac3e-802dccb6d79d-kube-api-access-cz7hd\") pod \"console-7c479d9b44-bctf4\" (UID: \"db081b6b-faa0-4113-ac3e-802dccb6d79d\") " pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:13.017452 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:13.017407    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:13.167829 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:13.167801    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-7c479d9b44-bctf4"]
Jul 16 21:58:13.170258 ip-10-0-129-202 kubenswrapper[2585]: W0716 21:58:13.170221    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb081b6b_faa0_4113_ac3e_802dccb6d79d.slice/crio-0a8537a39cc3991de0fbf67f09b518417f7b0306579dd7d0137a33bf6aa3ff40 WatchSource:0}: Error finding container 0a8537a39cc3991de0fbf67f09b518417f7b0306579dd7d0137a33bf6aa3ff40: Status 404 returned error can't find the container with id 0a8537a39cc3991de0fbf67f09b518417f7b0306579dd7d0137a33bf6aa3ff40
Jul 16 21:58:13.372989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:13.372886    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c479d9b44-bctf4" event={"ID":"db081b6b-faa0-4113-ac3e-802dccb6d79d","Type":"ContainerStarted","Data":"4719a12d7999a696c0e7269e9c13e6f0307256fef90e056f156419f8dd9138c5"}
Jul 16 21:58:13.372989 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:13.372940    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-7c479d9b44-bctf4" event={"ID":"db081b6b-faa0-4113-ac3e-802dccb6d79d","Type":"ContainerStarted","Data":"0a8537a39cc3991de0fbf67f09b518417f7b0306579dd7d0137a33bf6aa3ff40"}
Jul 16 21:58:13.409645 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:13.409586    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-7c479d9b44-bctf4" podStartSLOduration=1.409564686 podStartE2EDuration="1.409564686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 21:58:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 21:58:13.408208823 +0000 UTC m=+493.212632160" watchObservedRunningTime="2026-07-16 21:58:13.409564686 +0000 UTC m=+493.213988089"
Jul 16 21:58:19.137704 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:19.137665    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:23.018178 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.018132    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:23.018799 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.018201    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:23.023395 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.023365    2585 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:23.153744 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.153698    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:23.428292 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.428192    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-7c479d9b44-bctf4"
Jul 16 21:58:23.487081 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.487040    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f599b986-6ddpp"]
Jul 16 21:58:23.597603 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.597559    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:23.700026 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.699935    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:23.766448 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:23.766409    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:29.299390 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:29.299353    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve/kserve-controller-manager-84cf6d888d-gttm2"
Jul 16 21:58:30.362542 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:30.362494    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:30.481419 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:30.481377    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:31.403060 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:31.403018    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:31.473463 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:31.473416    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 21:58:48.510088 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.510019    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-55f599b986-6ddpp" podUID="9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" containerName="console" containerID="cri-o://b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d" gracePeriod=15
Jul 16 21:58:48.761526 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.761455    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:58:48.788561 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.788516    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hc2zg\" (UniqueName: \"kubernetes.io/projected/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-kube-api-access-hc2zg\") pod \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") "
Jul 16 21:58:48.788561 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.788567    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-oauth-config\") pod \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") "
Jul 16 21:58:48.789055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.788624    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-serving-cert\") pod \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") "
Jul 16 21:58:48.789055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.788647    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-trusted-ca-bundle\") pod \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") "
Jul 16 21:58:48.789055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.788666    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-oauth-serving-cert\") pod \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") "
Jul 16 21:58:48.789055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.788712    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-service-ca\") pod \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") "
Jul 16 21:58:48.789055 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.788742    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-config\") pod \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\" (UID: \"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb\") "
Jul 16 21:58:48.789348 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.789192    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" (UID: "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:58:48.789348 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.789199    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-trusted-ca-bundle" (OuterVolumeSpecName: "trusted-ca-bundle") pod "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" (UID: "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:58:48.789456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.789229    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-service-ca" (OuterVolumeSpecName: "service-ca") pod "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" (UID: "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:58:48.789456 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.789404    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-config" (OuterVolumeSpecName: "console-config") pod "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" (UID: "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 21:58:48.790867 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.790842    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" (UID: "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:58:48.790976 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.790947    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" (UID: "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 21:58:48.790976 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.790955    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-kube-api-access-hc2zg" (OuterVolumeSpecName: "kube-api-access-hc2zg") pod "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" (UID: "9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb"). InnerVolumeSpecName "kube-api-access-hc2zg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 21:58:48.889569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.889525    2585 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:58:48.889569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.889563    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hc2zg\" (UniqueName: \"kubernetes.io/projected/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-kube-api-access-hc2zg\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:58:48.889569 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.889575    2585 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-oauth-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:58:48.889844 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.889585    2585 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-console-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:58:48.889844 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.889596    2585 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-trusted-ca-bundle\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:58:48.889844 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.889604    2585 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-oauth-serving-cert\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:58:48.889844 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:48.889613    2585 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb-service-ca\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 21:58:49.562303 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.562265    2585 generic.go:356] "Generic (PLEG): container finished" podID="9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" containerID="b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d" exitCode=0
Jul 16 21:58:49.562836 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.562325    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-55f599b986-6ddpp"
Jul 16 21:58:49.562836 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.562347    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f599b986-6ddpp" event={"ID":"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb","Type":"ContainerDied","Data":"b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d"}
Jul 16 21:58:49.562836 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.562400    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-55f599b986-6ddpp" event={"ID":"9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb","Type":"ContainerDied","Data":"a05e3c8152b463775995701c6718973ae562ae968da131d3cb4964d9a1bfe3ec"}
Jul 16 21:58:49.562836 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.562423    2585 scope.go:117] "RemoveContainer" containerID="b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d"
Jul 16 21:58:49.571580 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.571557    2585 scope.go:117] "RemoveContainer" containerID="b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d"
Jul 16 21:58:49.571898 ip-10-0-129-202 kubenswrapper[2585]: E0716 21:58:49.571877    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d\": container with ID starting with b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d not found: ID does not exist" containerID="b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d"
Jul 16 21:58:49.571992 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.571907    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d"} err="failed to get container status \"b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d\": rpc error: code = NotFound desc = could not find container \"b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d\": container with ID starting with b7b80c1c00cb8dd91bf29957bd76719c7688a9847707f7598c7eae987e8c188d not found: ID does not exist"
Jul 16 21:58:49.590172 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.590131    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-55f599b986-6ddpp"]
Jul 16 21:58:49.608339 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:49.608300    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-55f599b986-6ddpp"]
Jul 16 21:58:50.828134 ip-10-0-129-202 kubenswrapper[2585]: I0716 21:58:50.828080    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" path="/var/lib/kubelet/pods/9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb/volumes"
Jul 16 22:00:00.672187 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:00.672133    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:00:00.674732 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:00.674709    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:00:05.232902 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:05.232857    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:05.347308 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:05.347260    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:16.699237 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.699197    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:16.738019 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.737976    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"]
Jul 16 22:00:16.738518 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.738497    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" containerName="console"
Jul 16 22:00:16.738608 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.738542    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" containerName="console"
Jul 16 22:00:16.738658 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.738642    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ef7a0b3-0ede-4e02-963e-a60ac6c70fbb" containerName="console"
Jul 16 22:00:16.741946 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.741924    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:16.744504 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.744479    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d9a7f-predictor-serving-cert\""
Jul 16 22:00:16.744680 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.744641    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"openshift-service-ca.crt\""
Jul 16 22:00:16.744804 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.744783    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"default-dockercfg-65f4r\""
Jul 16 22:00:16.745388 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.745365    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-d9a7f-kube-rbac-proxy-sar-config\""
Jul 16 22:00:16.745501 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.745404    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"kube-root-ca.crt\""
Jul 16 22:00:16.753706 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.753680    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"]
Jul 16 22:00:16.901012 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.900973    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pmdl\" (UniqueName: \"kubernetes.io/projected/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-api-access-9pmdl\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:16.901218 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.901056    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:16.901218 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:16.901093    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.002499 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.002460    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.002499 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.002503    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.002739 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.002558    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pmdl\" (UniqueName: \"kubernetes.io/projected/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-api-access-9pmdl\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.002739 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:00:17.002671    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-serving-cert: secret "success-200-isvc-d9a7f-predictor-serving-cert" not found
Jul 16 22:00:17.002816 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:00:17.002740    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls podName:d1f09258-3f0e-4e16-8890-939e6a4d7c90 nodeName:}" failed. No retries permitted until 2026-07-16 22:00:17.502724021 +0000 UTC m=+617.307147328 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls") pod "success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" (UID: "d1f09258-3f0e-4e16-8890-939e6a4d7c90") : secret "success-200-isvc-d9a7f-predictor-serving-cert" not found
Jul 16 22:00:17.003165 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.003147    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.012951 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.012919    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pmdl\" (UniqueName: \"kubernetes.io/projected/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-api-access-9pmdl\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.250258 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.250218    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"]
Jul 16 22:00:17.254052 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.253987    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.258067 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.258042    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d9a7f-predictor-serving-cert\""
Jul 16 22:00:17.258306 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.258288    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-d9a7f-kube-rbac-proxy-sar-config\""
Jul 16 22:00:17.279611 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.279566    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"]
Jul 16 22:00:17.405648 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.405606    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-proxy-tls\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.405860 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.405670    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.405860 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.405781    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ns246\" (UniqueName: \"kubernetes.io/projected/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-api-access-ns246\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.506491 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.506402    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.506641 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.506490    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-proxy-tls\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.506641 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.506534    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.506641 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.506571    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ns246\" (UniqueName: \"kubernetes.io/projected/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-api-access-ns246\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.507351 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.507328    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.509107 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.509059    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-proxy-tls\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.509213 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.509130    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls\") pod \"success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") " pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.520469 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.520434    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ns246\" (UniqueName: \"kubernetes.io/projected/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-api-access-ns246\") pod \"error-404-isvc-d9a7f-predictor-54f6744545-sj44g\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") " pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.565129 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.565077    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:17.653912 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.653874    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:17.727057 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.727029    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"]
Jul 16 22:00:17.729600 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:00:17.729569    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a49da02_ab1e_47aa_8a44_de3f7311e4d3.slice/crio-3940afe02a6c4ec69dc0324dba4fa22216feb771469ddc5bd10edcf323d3a1b5 WatchSource:0}: Error finding container 3940afe02a6c4ec69dc0324dba4fa22216feb771469ddc5bd10edcf323d3a1b5: Status 404 returned error can't find the container with id 3940afe02a6c4ec69dc0324dba4fa22216feb771469ddc5bd10edcf323d3a1b5
Jul 16 22:00:17.810945 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:17.810919    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"]
Jul 16 22:00:17.812708 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:00:17.812682    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1f09258_3f0e_4e16_8890_939e6a4d7c90.slice/crio-baaca026c1237b10d0a067bad3e37c52cc27ac05e6a071cb8af2194fe5875a95 WatchSource:0}: Error finding container baaca026c1237b10d0a067bad3e37c52cc27ac05e6a071cb8af2194fe5875a95: Status 404 returned error can't find the container with id baaca026c1237b10d0a067bad3e37c52cc27ac05e6a071cb8af2194fe5875a95
Jul 16 22:00:18.003804 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:18.003764    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" event={"ID":"9a49da02-ab1e-47aa-8a44-de3f7311e4d3","Type":"ContainerStarted","Data":"3940afe02a6c4ec69dc0324dba4fa22216feb771469ddc5bd10edcf323d3a1b5"}
Jul 16 22:00:18.005006 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:18.004969    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" event={"ID":"d1f09258-3f0e-4e16-8890-939e6a4d7c90","Type":"ContainerStarted","Data":"baaca026c1237b10d0a067bad3e37c52cc27ac05e6a071cb8af2194fe5875a95"}
Jul 16 22:00:18.263002 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:18.262968    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:18.330304 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:18.330269    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:19.115446 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:19.115389    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:24.520141 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:24.519872    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:24.578059 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:24.578018    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:28.544760 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:28.544713    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:00:33.467977 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:33.467949    2585 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 16 22:00:34.135839 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:34.135794    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" event={"ID":"d1f09258-3f0e-4e16-8890-939e6a4d7c90","Type":"ContainerStarted","Data":"313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424"}
Jul 16 22:00:34.137913 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:34.137872    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" event={"ID":"9a49da02-ab1e-47aa-8a44-de3f7311e4d3","Type":"ContainerStarted","Data":"4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0"}
Jul 16 22:00:36.150378 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:36.150332    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" event={"ID":"9a49da02-ab1e-47aa-8a44-de3f7311e4d3","Type":"ContainerStarted","Data":"65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725"}
Jul 16 22:00:36.150862 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:36.150450    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:36.152124 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:36.152080    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" event={"ID":"d1f09258-3f0e-4e16-8890-939e6a4d7c90","Type":"ContainerStarted","Data":"2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8"}
Jul 16 22:00:36.152313 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:36.152291    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:36.152407 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:36.152347    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:36.153998 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:36.153967    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 16 22:00:36.171835 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:36.171779    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podStartSLOduration=0.930131256 podStartE2EDuration="19.171761548s" totalImagesPullingTime="18.241630292s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:36.169869641 +0000 UTC m=+635.974292968" watchObservedRunningTime="2026-07-16 22:00:36.171761548 +0000 UTC m=+635.976184874"
Jul 16 22:00:36.189056 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:36.188998    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podStartSLOduration=2.118960444 podStartE2EDuration="20.188967853s" totalImagesPullingTime="18.070007409s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:00:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:00:36.187664763 +0000 UTC m=+635.992088089" watchObservedRunningTime="2026-07-16 22:00:36.188967853 +0000 UTC m=+635.993391178"
Jul 16 22:00:37.157244 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:37.157208    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:37.157244 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:37.157208    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 16 22:00:37.158801 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:37.158767    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 16 22:00:38.162620 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:38.162578    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 16 22:00:42.162479 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:42.162445    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:00:42.162935 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:42.162905    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 16 22:00:43.167138 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:43.167085    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:00:43.167640 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:43.167610    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 16 22:00:52.163585 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:52.163539    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 16 22:00:53.167999 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:00:53.167956    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 16 22:01:02.163188 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:02.163136    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 16 22:01:03.167681 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:03.167639    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 16 22:01:12.163369 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:12.163320    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 16 22:01:13.168388 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:13.168345    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 16 22:01:22.164313 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:22.164273    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:01:23.168804 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:23.168764    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:01:36.504421 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:36.504384    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:41.354741 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:41.354705    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:51.117517 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.117482    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:51.265289 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.265246    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"]
Jul 16 22:01:51.265556 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.265530    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container" containerID="cri-o://313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424" gracePeriod=30
Jul 16 22:01:51.265637 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.265595    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kube-rbac-proxy" containerID="cri-o://2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8" gracePeriod=30
Jul 16 22:01:51.350407 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.350360    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:51.367267 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.367222    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"]
Jul 16 22:01:51.367492 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.367464    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container" containerID="cri-o://4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0" gracePeriod=30
Jul 16 22:01:51.367594 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.367519    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kube-rbac-proxy" containerID="cri-o://65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725" gracePeriod=30
Jul 16 22:01:51.399005 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.398964    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"]
Jul 16 22:01:51.402952 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.402924    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.405585 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.405561    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-69517-predictor-serving-cert\""
Jul 16 22:01:51.408859 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.408832    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-69517-kube-rbac-proxy-sar-config\""
Jul 16 22:01:51.415251 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.415225    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"]
Jul 16 22:01:51.460933 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.460890    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:51.489277 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.489233    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"]
Jul 16 22:01:51.492905 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.492879    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.495524 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.495495    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-69517-kube-rbac-proxy-sar-config\""
Jul 16 22:01:51.495686 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.495497    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-69517-predictor-serving-cert\""
Jul 16 22:01:51.504600 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.504572    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"]
Jul 16 22:01:51.527011 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.526976    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b29823-fd41-4147-959f-e09b96ad0567-proxy-tls\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.527215 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.527115    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmc9q\" (UniqueName: \"kubernetes.io/projected/d6b29823-fd41-4147-959f-e09b96ad0567-kube-api-access-nmc9q\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.527215 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.527173    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b29823-fd41-4147-959f-e09b96ad0567-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.537223 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.537187    2585 generic.go:356] "Generic (PLEG): container finished" podID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerID="65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725" exitCode=2
Jul 16 22:01:51.537415 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.537252    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" event={"ID":"9a49da02-ab1e-47aa-8a44-de3f7311e4d3","Type":"ContainerDied","Data":"65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725"}
Jul 16 22:01:51.539180 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.539155    2585 generic.go:356] "Generic (PLEG): container finished" podID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerID="2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8" exitCode=2
Jul 16 22:01:51.539321 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.539222    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" event={"ID":"d1f09258-3f0e-4e16-8890-939e6a4d7c90","Type":"ContainerDied","Data":"2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8"}
Jul 16 22:01:51.544673 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.544648    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:51.628786 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.628684    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8qwwr\" (UniqueName: \"kubernetes.io/projected/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-api-access-8qwwr\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.628786 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.628770    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmc9q\" (UniqueName: \"kubernetes.io/projected/d6b29823-fd41-4147-959f-e09b96ad0567-kube-api-access-nmc9q\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.629558 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.628813    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b29823-fd41-4147-959f-e09b96ad0567-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.629558 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.629349    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b29823-fd41-4147-959f-e09b96ad0567-proxy-tls\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.629558 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.629446    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.629558 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.629488    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.629796 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.629715    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b29823-fd41-4147-959f-e09b96ad0567-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.632047 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.632018    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b29823-fd41-4147-959f-e09b96ad0567-proxy-tls\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.638354 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.638330    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmc9q\" (UniqueName: \"kubernetes.io/projected/d6b29823-fd41-4147-959f-e09b96ad0567-kube-api-access-nmc9q\") pod \"success-200-isvc-69517-predictor-85bc9bd758-gtdkw\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") " pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.674890 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.674857    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:51.715302 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.715252    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:51.730901 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.730867    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.731054 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.730909    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8qwwr\" (UniqueName: \"kubernetes.io/projected/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-api-access-8qwwr\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.731054 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.731015    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.731054 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:01:51.731036    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-69517-predictor-serving-cert: secret "error-404-isvc-69517-predictor-serving-cert" not found
Jul 16 22:01:51.731250 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:01:51.731132    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls podName:36437163-7ef1-4122-93fc-808ae5ae10a1 nodeName:}" failed. No retries permitted until 2026-07-16 22:01:52.231110901 +0000 UTC m=+712.035534218 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls") pod "error-404-isvc-69517-predictor-86957765c5-22c7r" (UID: "36437163-7ef1-4122-93fc-808ae5ae10a1") : secret "error-404-isvc-69517-predictor-serving-cert" not found
Jul 16 22:01:51.731669 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.731634    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.740274 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.740236    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8qwwr\" (UniqueName: \"kubernetes.io/projected/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-api-access-8qwwr\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:51.827284 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.827242    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:01:51.865670 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:51.865642    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"]
Jul 16 22:01:51.867905 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:01:51.867873    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b29823_fd41_4147_959f_e09b96ad0567.slice/crio-fa6fcc05dfbddc58627c65800b413337317e9c181569c222d0159706ae577f20 WatchSource:0}: Error finding container fa6fcc05dfbddc58627c65800b413337317e9c181569c222d0159706ae577f20: Status 404 returned error can't find the container with id fa6fcc05dfbddc58627c65800b413337317e9c181569c222d0159706ae577f20
Jul 16 22:01:52.157903 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.157788    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.39:8643/healthz\": dial tcp 10.134.0.39:8643: connect: connection refused"
Jul 16 22:01:52.162883 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.162845    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.39:8080: connect: connection refused"
Jul 16 22:01:52.236552 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.236506    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:52.239057 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.239024    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls\") pod \"error-404-isvc-69517-predictor-86957765c5-22c7r\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") " pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:52.405261 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.405168    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:52.547212 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.546785    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" event={"ID":"d6b29823-fd41-4147-959f-e09b96ad0567","Type":"ContainerStarted","Data":"ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4"}
Jul 16 22:01:52.547212 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.546876    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" event={"ID":"d6b29823-fd41-4147-959f-e09b96ad0567","Type":"ContainerStarted","Data":"44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38"}
Jul 16 22:01:52.547212 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.546909    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" event={"ID":"d6b29823-fd41-4147-959f-e09b96ad0567","Type":"ContainerStarted","Data":"fa6fcc05dfbddc58627c65800b413337317e9c181569c222d0159706ae577f20"}
Jul 16 22:01:52.547675 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.547409    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:52.547675 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.547488    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:52.548721 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.548687    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:01:52.549774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.549720    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"]
Jul 16 22:01:52.552352 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:01:52.552322    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36437163_7ef1_4122_93fc_808ae5ae10a1.slice/crio-f3c0292380a3200f342ecdd1d64e66845d9b85ffa47542154fe24c869d95fee4 WatchSource:0}: Error finding container f3c0292380a3200f342ecdd1d64e66845d9b85ffa47542154fe24c869d95fee4: Status 404 returned error can't find the container with id f3c0292380a3200f342ecdd1d64e66845d9b85ffa47542154fe24c869d95fee4
Jul 16 22:01:52.566074 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:52.566024    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podStartSLOduration=1.5660078739999999 podStartE2EDuration="1.566007874s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:52.56435902 +0000 UTC m=+712.368782337" watchObservedRunningTime="2026-07-16 22:01:52.566007874 +0000 UTC m=+712.370431199"
Jul 16 22:01:53.163710 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:53.163655    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.40:8643/healthz\": dial tcp 10.134.0.40:8643: connect: connection refused"
Jul 16 22:01:53.168077 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:53.168044    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.40:8080: connect: connection refused"
Jul 16 22:01:53.558407 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:53.558366    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" event={"ID":"36437163-7ef1-4122-93fc-808ae5ae10a1","Type":"ContainerStarted","Data":"f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b"}
Jul 16 22:01:53.558627 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:53.558478    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" event={"ID":"36437163-7ef1-4122-93fc-808ae5ae10a1","Type":"ContainerStarted","Data":"e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085"}
Jul 16 22:01:53.558627 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:53.558513    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" event={"ID":"36437163-7ef1-4122-93fc-808ae5ae10a1","Type":"ContainerStarted","Data":"f3c0292380a3200f342ecdd1d64e66845d9b85ffa47542154fe24c869d95fee4"}
Jul 16 22:01:53.558789 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:53.558763    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:01:53.581082 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:53.581023    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podStartSLOduration=2.581003528 podStartE2EDuration="2.581003528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:01:53.579552021 +0000 UTC m=+713.383975371" watchObservedRunningTime="2026-07-16 22:01:53.581003528 +0000 UTC m=+713.385426854"
Jul 16 22:01:54.564489 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:54.564439    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:01:54.564489 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:54.564470    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:54.565042 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:54.564558    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:01:54.565655 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:54.565630    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 16 22:01:55.241312 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.241284    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:01:55.365402 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.365364    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ns246\" (UniqueName: \"kubernetes.io/projected/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-api-access-ns246\") pod \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") "
Jul 16 22:01:55.365607 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.365471    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-proxy-tls\") pod \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") "
Jul 16 22:01:55.365607 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.365501    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-rbac-proxy-sar-config\") pod \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\" (UID: \"9a49da02-ab1e-47aa-8a44-de3f7311e4d3\") "
Jul 16 22:01:55.365895 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.365866    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "9a49da02-ab1e-47aa-8a44-de3f7311e4d3" (UID: "9a49da02-ab1e-47aa-8a44-de3f7311e4d3"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:01:55.367788 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.367756    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "9a49da02-ab1e-47aa-8a44-de3f7311e4d3" (UID: "9a49da02-ab1e-47aa-8a44-de3f7311e4d3"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:01:55.367910 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.367824    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-api-access-ns246" (OuterVolumeSpecName: "kube-api-access-ns246") pod "9a49da02-ab1e-47aa-8a44-de3f7311e4d3" (UID: "9a49da02-ab1e-47aa-8a44-de3f7311e4d3"). InnerVolumeSpecName "kube-api-access-ns246". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:01:55.466875 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.466833    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:01:55.466875 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.466866    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:01:55.466875 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.466881    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ns246\" (UniqueName: \"kubernetes.io/projected/9a49da02-ab1e-47aa-8a44-de3f7311e4d3-kube-api-access-ns246\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:01:55.520042 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.520011    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:01:55.572863 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.572751    2585 generic.go:356] "Generic (PLEG): container finished" podID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerID="4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0" exitCode=0
Jul 16 22:01:55.572863 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.572846    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"
Jul 16 22:01:55.573424 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.572844    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" event={"ID":"9a49da02-ab1e-47aa-8a44-de3f7311e4d3","Type":"ContainerDied","Data":"4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0"}
Jul 16 22:01:55.573424 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.572961    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g" event={"ID":"9a49da02-ab1e-47aa-8a44-de3f7311e4d3","Type":"ContainerDied","Data":"3940afe02a6c4ec69dc0324dba4fa22216feb771469ddc5bd10edcf323d3a1b5"}
Jul 16 22:01:55.573424 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.572991    2585 scope.go:117] "RemoveContainer" containerID="65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725"
Jul 16 22:01:55.574662 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.574635    2585 generic.go:356] "Generic (PLEG): container finished" podID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerID="313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424" exitCode=0
Jul 16 22:01:55.574805 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.574678    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"
Jul 16 22:01:55.574805 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.574722    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" event={"ID":"d1f09258-3f0e-4e16-8890-939e6a4d7c90","Type":"ContainerDied","Data":"313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424"}
Jul 16 22:01:55.574805 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.574773    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq" event={"ID":"d1f09258-3f0e-4e16-8890-939e6a4d7c90","Type":"ContainerDied","Data":"baaca026c1237b10d0a067bad3e37c52cc27ac05e6a071cb8af2194fe5875a95"}
Jul 16 22:01:55.575134 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.575091    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 16 22:01:55.582769 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.582750    2585 scope.go:117] "RemoveContainer" containerID="4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0"
Jul 16 22:01:55.591270 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.591250    2585 scope.go:117] "RemoveContainer" containerID="65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725"
Jul 16 22:01:55.591580 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:01:55.591557    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725\": container with ID starting with 65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725 not found: ID does not exist" containerID="65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725"
Jul 16 22:01:55.591673 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.591594    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725"} err="failed to get container status \"65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725\": rpc error: code = NotFound desc = could not find container \"65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725\": container with ID starting with 65d936e6c1f3856edffbdcd37a34e7e0b15670ab985a663d4df9fc77c243d725 not found: ID does not exist"
Jul 16 22:01:55.591673 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.591619    2585 scope.go:117] "RemoveContainer" containerID="4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0"
Jul 16 22:01:55.591870 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:01:55.591854    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0\": container with ID starting with 4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0 not found: ID does not exist" containerID="4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0"
Jul 16 22:01:55.591912 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.591877    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0"} err="failed to get container status \"4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0\": rpc error: code = NotFound desc = could not find container \"4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0\": container with ID starting with 4e4b350c22cead38063f87234999afad0d996de0788bf6642e9f050a6cd460e0 not found: ID does not exist"
Jul 16 22:01:55.591912 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.591891    2585 scope.go:117] "RemoveContainer" containerID="2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8"
Jul 16 22:01:55.597219 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.597191    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"]
Jul 16 22:01:55.601451 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.601421    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-d9a7f-predictor-54f6744545-sj44g"]
Jul 16 22:01:55.601801 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.601773    2585 scope.go:117] "RemoveContainer" containerID="313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424"
Jul 16 22:01:55.611151 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.611128    2585 scope.go:117] "RemoveContainer" containerID="2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8"
Jul 16 22:01:55.611521 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:01:55.611494    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8\": container with ID starting with 2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8 not found: ID does not exist" containerID="2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8"
Jul 16 22:01:55.611595 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.611537    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8"} err="failed to get container status \"2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8\": rpc error: code = NotFound desc = could not find container \"2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8\": container with ID starting with 2947987e3624d80d2aef977239bb21a802a99470069919136d7ec49159bd53f8 not found: ID does not exist"
Jul 16 22:01:55.611595 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.611565    2585 scope.go:117] "RemoveContainer" containerID="313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424"
Jul 16 22:01:55.611890 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:01:55.611870    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424\": container with ID starting with 313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424 not found: ID does not exist" containerID="313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424"
Jul 16 22:01:55.611939 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.611896    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424"} err="failed to get container status \"313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424\": rpc error: code = NotFound desc = could not find container \"313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424\": container with ID starting with 313520068b16a0ce5b694839deff778e8e753315b9b0042ac544bbb021457424 not found: ID does not exist"
Jul 16 22:01:55.668197 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.668167    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-rbac-proxy-sar-config\") pod \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") "
Jul 16 22:01:55.668357 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.668210    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9pmdl\" (UniqueName: \"kubernetes.io/projected/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-api-access-9pmdl\") pod \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") "
Jul 16 22:01:55.668357 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.668253    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls\") pod \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\" (UID: \"d1f09258-3f0e-4e16-8890-939e6a4d7c90\") "
Jul 16 22:01:55.668596 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.668571    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d1f09258-3f0e-4e16-8890-939e6a4d7c90" (UID: "d1f09258-3f0e-4e16-8890-939e6a4d7c90"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:01:55.670443 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.670415    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d1f09258-3f0e-4e16-8890-939e6a4d7c90" (UID: "d1f09258-3f0e-4e16-8890-939e6a4d7c90"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:01:55.670528 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.670456    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-api-access-9pmdl" (OuterVolumeSpecName: "kube-api-access-9pmdl") pod "d1f09258-3f0e-4e16-8890-939e6a4d7c90" (UID: "d1f09258-3f0e-4e16-8890-939e6a4d7c90"). InnerVolumeSpecName "kube-api-access-9pmdl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:01:55.769204 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.769153    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d1f09258-3f0e-4e16-8890-939e6a4d7c90-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:01:55.769204 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.769197    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:01:55.769204 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.769211    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pmdl\" (UniqueName: \"kubernetes.io/projected/d1f09258-3f0e-4e16-8890-939e6a4d7c90-kube-api-access-9pmdl\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:01:55.905573 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.905531    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"]
Jul 16 22:01:55.911881 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:55.911830    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-d9a7f-predictor-956dc5df6-ckrlq"]
Jul 16 22:01:56.827320 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:56.827283    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" path="/var/lib/kubelet/pods/9a49da02-ab1e-47aa-8a44-de3f7311e4d3/volumes"
Jul 16 22:01:56.827763 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:56.827749    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" path="/var/lib/kubelet/pods/d1f09258-3f0e-4e16-8890-939e6a4d7c90/volumes"
Jul 16 22:01:59.569195 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:59.569159    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:01:59.569817 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:01:59.569777    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:02:00.579642 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:00.579611    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:02:00.580152 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:00.580123    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 16 22:02:09.569687 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:09.569641    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:02:10.580268 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:10.580227    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 16 22:02:19.569950 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:19.569906    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:02:20.580148 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:20.580082    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 16 22:02:26.871706 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:26.871652    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:27.013749 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:27.013714    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:29.569717 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:29.569674    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:02:30.580867 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:30.580808    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.42:8080: connect: connection refused"
Jul 16 22:02:33.747811 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:33.747773    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:37.002302 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.002247    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:37.239168 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.239108    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:37.263702 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.263609    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"]
Jul 16 22:02:37.264329 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264311    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container"
Jul 16 22:02:37.264398 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264333    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container"
Jul 16 22:02:37.264398 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264348    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kube-rbac-proxy"
Jul 16 22:02:37.264398 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264358    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kube-rbac-proxy"
Jul 16 22:02:37.264398 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264395    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container"
Jul 16 22:02:37.264526 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264404    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container"
Jul 16 22:02:37.264526 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264416    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kube-rbac-proxy"
Jul 16 22:02:37.264526 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264425    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kube-rbac-proxy"
Jul 16 22:02:37.264526 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264516    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kserve-container"
Jul 16 22:02:37.264650 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264530    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kserve-container"
Jul 16 22:02:37.264650 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264543    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1f09258-3f0e-4e16-8890-939e6a4d7c90" containerName="kube-rbac-proxy"
Jul 16 22:02:37.264650 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.264553    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="9a49da02-ab1e-47aa-8a44-de3f7311e4d3" containerName="kube-rbac-proxy"
Jul 16 22:02:37.268459 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.268427    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.271531 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.271504    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-77f35-predictor-serving-cert\""
Jul 16 22:02:37.271675 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.271505    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-77f35-kube-rbac-proxy-sar-config\""
Jul 16 22:02:37.287301 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.287255    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"]
Jul 16 22:02:37.346296 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.346264    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:37.357707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.357672    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-proxy-tls\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.357885 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.357716    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.357885 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.357840    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-md4gc\" (UniqueName: \"kubernetes.io/projected/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-api-access-md4gc\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.419020 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.418983    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"]
Jul 16 22:02:37.422856 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.422830    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.426030 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.425998    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-77f35-kube-rbac-proxy-sar-config\""
Jul 16 22:02:37.428987 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.428961    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-77f35-predictor-serving-cert\""
Jul 16 22:02:37.453232 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.453197    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"]
Jul 16 22:02:37.458616 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.458578    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-proxy-tls\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.458616 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.458616    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.458887 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.458668    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-md4gc\" (UniqueName: \"kubernetes.io/projected/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-api-access-md4gc\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.459387 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.459360    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.461224 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.461193    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-proxy-tls\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.499039 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.498971    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-md4gc\" (UniqueName: \"kubernetes.io/projected/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-api-access-md4gc\") pod \"success-200-isvc-77f35-predictor-766f77d998-qfjbd\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") " pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.503173 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.503143    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:37.559520 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.559436    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqqm2\" (UniqueName: \"kubernetes.io/projected/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-api-access-sqqm2\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.559520 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.559480    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.559703 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.559573    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.580650 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.580606    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:37.660493 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.660397    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqqm2\" (UniqueName: \"kubernetes.io/projected/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-api-access-sqqm2\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.660493 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.660444    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.660719 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.660535    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.661311 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:02:37.660917    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-77f35-predictor-serving-cert: secret "error-404-isvc-77f35-predictor-serving-cert" not found
Jul 16 22:02:37.661311 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:02:37.661045    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls podName:b2a4d504-f247-461b-9f65-b8a83e2837d6 nodeName:}" failed. No retries permitted until 2026-07-16 22:02:38.161020438 +0000 UTC m=+757.965443752 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls") pod "error-404-isvc-77f35-predictor-558485f7c5-gvp7z" (UID: "b2a4d504-f247-461b-9f65-b8a83e2837d6") : secret "error-404-isvc-77f35-predictor-serving-cert" not found
Jul 16 22:02:37.661311 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.661304    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.678779 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.678735    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqqm2\" (UniqueName: \"kubernetes.io/projected/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-api-access-sqqm2\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:37.716385 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.716341    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:37.734204 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.734169    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"]
Jul 16 22:02:37.736079 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:02:37.736047    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8832ac4b_a3b2_4a46_82ca_c9049562d2e7.slice/crio-a129622b4fbcceb766f0de20c232772f5d864ed91a3525f331d918984d3e0f53 WatchSource:0}: Error finding container a129622b4fbcceb766f0de20c232772f5d864ed91a3525f331d918984d3e0f53: Status 404 returned error can't find the container with id a129622b4fbcceb766f0de20c232772f5d864ed91a3525f331d918984d3e0f53
Jul 16 22:02:37.804583 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.804548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" event={"ID":"8832ac4b-a3b2-4a46-82ca-c9049562d2e7","Type":"ContainerStarted","Data":"bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50"}
Jul 16 22:02:37.804703 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.804616    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" event={"ID":"8832ac4b-a3b2-4a46-82ca-c9049562d2e7","Type":"ContainerStarted","Data":"a129622b4fbcceb766f0de20c232772f5d864ed91a3525f331d918984d3e0f53"}
Jul 16 22:02:37.812607 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:37.812532    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:02:38.165273 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.165155    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:38.167649 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.167626    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls\") pod \"error-404-isvc-77f35-predictor-558485f7c5-gvp7z\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") " pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:38.334309 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.334263    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:38.476786 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.476757    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"]
Jul 16 22:02:38.478585 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:02:38.478553    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2a4d504_f247_461b_9f65_b8a83e2837d6.slice/crio-f0b5595fafea8d38ab59dd61f25569449609c0e658506cb08280c5e3df432611 WatchSource:0}: Error finding container f0b5595fafea8d38ab59dd61f25569449609c0e658506cb08280c5e3df432611: Status 404 returned error can't find the container with id f0b5595fafea8d38ab59dd61f25569449609c0e658506cb08280c5e3df432611
Jul 16 22:02:38.814365 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.814316    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" event={"ID":"8832ac4b-a3b2-4a46-82ca-c9049562d2e7","Type":"ContainerStarted","Data":"f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50"}
Jul 16 22:02:38.815461 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.815349    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:38.815461 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.815414    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:38.817552 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.817261    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 16 22:02:38.818600 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.818233    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" event={"ID":"b2a4d504-f247-461b-9f65-b8a83e2837d6","Type":"ContainerStarted","Data":"15ec9475679c90cdc8c33acda16615274d8533284db6197bfd71ff3f26a2c43a"}
Jul 16 22:02:38.818600 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.818313    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:38.818600 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.818346    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" event={"ID":"b2a4d504-f247-461b-9f65-b8a83e2837d6","Type":"ContainerStarted","Data":"235fda80d09acdf0d533f075db44068bba4e9e0b4837256161dc4f94d76b5a93"}
Jul 16 22:02:38.818600 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.818376    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:38.818600 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.818401    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" event={"ID":"b2a4d504-f247-461b-9f65-b8a83e2837d6","Type":"ContainerStarted","Data":"f0b5595fafea8d38ab59dd61f25569449609c0e658506cb08280c5e3df432611"}
Jul 16 22:02:38.819703 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.819672    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 16 22:02:38.837363 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.837309    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podStartSLOduration=1.837290741 podStartE2EDuration="1.837290741s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:38.835000025 +0000 UTC m=+758.639423379" watchObservedRunningTime="2026-07-16 22:02:38.837290741 +0000 UTC m=+758.641714067"
Jul 16 22:02:38.854909 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:38.854847    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podStartSLOduration=1.8548307240000002 podStartE2EDuration="1.854830724s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:02:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:02:38.852987627 +0000 UTC m=+758.657410953" watchObservedRunningTime="2026-07-16 22:02:38.854830724 +0000 UTC m=+758.659254051"
Jul 16 22:02:39.569751 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:39.569707    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:02:39.824628 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:39.824503    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 16 22:02:39.824628 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:39.824559    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 16 22:02:40.581978 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:40.581942    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:02:40.829186 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:40.829141    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 16 22:02:44.829215 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:44.829184    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:02:44.830870 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:44.830829    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 16 22:02:45.833508 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:45.833477    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:02:45.834010 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:45.833979    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 16 22:02:49.571280 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:49.571249    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:02:54.830368 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:54.830324    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 16 22:02:55.834401 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:02:55.834357    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 16 22:03:01.402878 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:01.402842    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:03:01.723973 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:01.723892    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:03:04.830378 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:04.830332    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 16 22:03:05.834173 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:05.834130    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 16 22:03:14.829758 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:14.829710    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 16 22:03:15.833951 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:15.833906    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 16 22:03:24.830625 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:24.830534    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 16 22:03:25.835456 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:25.835415    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:03:34.834032 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:34.834002    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:03:37.266328 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:37.266289    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:03:37.359656 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:37.359619    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:03:52.741291 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:52.741252    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:03:52.805129 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:03:52.805076    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:05:00.710967 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:05:00.710933    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:05:00.711458 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:05:00.710937    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:05:02.337479 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:05:02.337433    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:06:26.683219 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:06:26.683125    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:08:20.672288 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:08:20.672164    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:10:00.744843 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:10:00.744805    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:10:00.746088 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:10:00.746065    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:11:16.011148 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.011053    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:16.103989 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.103949    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"]
Jul 16 22:11:16.104349 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.104316    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" containerID="cri-o://44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38" gracePeriod=30
Jul 16 22:11:16.104464 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.104433    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kube-rbac-proxy" containerID="cri-o://ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4" gracePeriod=30
Jul 16 22:11:16.125466 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.125431    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:16.233072 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.233019    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:16.234879 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.234849    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"]
Jul 16 22:11:16.240001 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.239646    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.242925 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.242891    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1ab93-predictor-serving-cert\""
Jul 16 22:11:16.242925 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.242910    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-1ab93-kube-rbac-proxy-sar-config\""
Jul 16 22:11:16.254093 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.254057    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"]
Jul 16 22:11:16.254395 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.254366    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container" containerID="cri-o://e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085" gracePeriod=30
Jul 16 22:11:16.254478 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.254428    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kube-rbac-proxy" containerID="cri-o://f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b" gracePeriod=30
Jul 16 22:11:16.256908 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.256878    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"]
Jul 16 22:11:16.285628 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.285594    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.285821 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.285645    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.285821 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.285749    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ww5c8\" (UniqueName: \"kubernetes.io/projected/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-api-access-ww5c8\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.311356 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.311320    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"]
Jul 16 22:11:16.315362 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.315332    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.318417 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.318375    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1ab93-predictor-serving-cert\""
Jul 16 22:11:16.318417 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.318378    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-1ab93-kube-rbac-proxy-sar-config\""
Jul 16 22:11:16.337186 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.337137    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"]
Jul 16 22:11:16.347296 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.347247    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:16.386756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.386723    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.386932 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.386779    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.386932 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.386822    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsl8b\" (UniqueName: \"kubernetes.io/projected/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-api-access-nsl8b\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.386932 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.386840    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.386932 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.386912    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-proxy-tls\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.387137 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:16.386949    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-serving-cert: secret "success-200-isvc-1ab93-predictor-serving-cert" not found
Jul 16 22:11:16.387137 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:16.387007    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls podName:890b57c2-207d-4f3b-ba55-5607db805fd0 nodeName:}" failed. No retries permitted until 2026-07-16 22:11:16.886991505 +0000 UTC m=+1276.691414809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls") pod "success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" (UID: "890b57c2-207d-4f3b-ba55-5607db805fd0") : secret "success-200-isvc-1ab93-predictor-serving-cert" not found
Jul 16 22:11:16.387137 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.386950    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ww5c8\" (UniqueName: \"kubernetes.io/projected/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-api-access-ww5c8\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.387606 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.387584    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.408841 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.408805    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ww5c8\" (UniqueName: \"kubernetes.io/projected/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-api-access-ww5c8\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.454669 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.454630    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:16.469657 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.469619    2585 generic.go:356] "Generic (PLEG): container finished" podID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerID="f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b" exitCode=2
Jul 16 22:11:16.469837 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.469685    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" event={"ID":"36437163-7ef1-4122-93fc-808ae5ae10a1","Type":"ContainerDied","Data":"f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b"}
Jul 16 22:11:16.471881 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.471847    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6b29823-fd41-4147-959f-e09b96ad0567" containerID="ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4" exitCode=2
Jul 16 22:11:16.472022 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.471890    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" event={"ID":"d6b29823-fd41-4147-959f-e09b96ad0567","Type":"ContainerDied","Data":"ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4"}
Jul 16 22:11:16.488452 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.488411    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-proxy-tls\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.488618 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.488594    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsl8b\" (UniqueName: \"kubernetes.io/projected/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-api-access-nsl8b\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.488677 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.488626    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.489474 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.489434    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.490991 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.490968    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-proxy-tls\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.499831 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.499801    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsl8b\" (UniqueName: \"kubernetes.io/projected/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-api-access-nsl8b\") pod \"error-404-isvc-1ab93-predictor-7cd97fc847-qltmd\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") " pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.554065 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.553946    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:16.626353 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.626316    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:16.777195 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.777165    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"]
Jul 16 22:11:16.777893 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:11:16.777861    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ce564fd_bfcf_4e1b_8430_7347fb64adaa.slice/crio-ade0638d02f8cb3a794bc95e1058518e023fbe1e3424fe443fa56bdd96a84b2f WatchSource:0}: Error finding container ade0638d02f8cb3a794bc95e1058518e023fbe1e3424fe443fa56bdd96a84b2f: Status 404 returned error can't find the container with id ade0638d02f8cb3a794bc95e1058518e023fbe1e3424fe443fa56bdd96a84b2f
Jul 16 22:11:16.893533 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.893490    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:16.896072 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:16.896036    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls\") pod \"success-200-isvc-1ab93-predictor-d659cd46b-gcpdg\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") " pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:17.155078 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.154964    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:17.317952 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.317916    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"]
Jul 16 22:11:17.320119 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:11:17.320071    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod890b57c2_207d_4f3b_ba55_5607db805fd0.slice/crio-a5f09b40236e1a7992f122724c8d7c9ba61dc46416b6966c7ab81be0113e2232 WatchSource:0}: Error finding container a5f09b40236e1a7992f122724c8d7c9ba61dc46416b6966c7ab81be0113e2232: Status 404 returned error can't find the container with id a5f09b40236e1a7992f122724c8d7c9ba61dc46416b6966c7ab81be0113e2232
Jul 16 22:11:17.480054 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.480007    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" event={"ID":"890b57c2-207d-4f3b-ba55-5607db805fd0","Type":"ContainerStarted","Data":"6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a"}
Jul 16 22:11:17.480290 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.480087    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" event={"ID":"890b57c2-207d-4f3b-ba55-5607db805fd0","Type":"ContainerStarted","Data":"031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e"}
Jul 16 22:11:17.480290 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.480146    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:17.480290 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.480176    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" event={"ID":"890b57c2-207d-4f3b-ba55-5607db805fd0","Type":"ContainerStarted","Data":"a5f09b40236e1a7992f122724c8d7c9ba61dc46416b6966c7ab81be0113e2232"}
Jul 16 22:11:17.482045 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.482016    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" event={"ID":"6ce564fd-bfcf-4e1b-8430-7347fb64adaa","Type":"ContainerStarted","Data":"c57afc52fa17aea0b267d35431d36c81a676a7f01bde9afe928216c9b0f18606"}
Jul 16 22:11:17.482195 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.482081    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:17.482195 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.482137    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" event={"ID":"6ce564fd-bfcf-4e1b-8430-7347fb64adaa","Type":"ContainerStarted","Data":"dc6f23338d1f18c00e1571c92bbeb87b4f0ea934e0e350228ca1e2d3e53198f7"}
Jul 16 22:11:17.482195 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.482162    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" event={"ID":"6ce564fd-bfcf-4e1b-8430-7347fb64adaa","Type":"ContainerStarted","Data":"ade0638d02f8cb3a794bc95e1058518e023fbe1e3424fe443fa56bdd96a84b2f"}
Jul 16 22:11:17.501557 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.501488    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podStartSLOduration=1.50146592 podStartE2EDuration="1.50146592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:11:17.499127474 +0000 UTC m=+1277.303550798" watchObservedRunningTime="2026-07-16 22:11:17.50146592 +0000 UTC m=+1277.305889247"
Jul 16 22:11:17.517601 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:17.517520    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podStartSLOduration=1.517495905 podStartE2EDuration="1.517495905s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:11:17.516940259 +0000 UTC m=+1277.321363588" watchObservedRunningTime="2026-07-16 22:11:17.517495905 +0000 UTC m=+1277.321919234"
Jul 16 22:11:18.488317 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:18.488282    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:18.488730 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:18.488353    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:18.489393 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:18.489360    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 16 22:11:18.489549 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:18.489524    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 16 22:11:18.731484 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:18.731448    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:19.493020 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:19.492968    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 16 22:11:19.493525 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:19.493092    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 16 22:11:19.565324 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:19.565278    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.41:8643/healthz\": dial tcp 10.134.0.41:8643: connect: connection refused"
Jul 16 22:11:19.569836 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:19.569787    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.41:8080: connect: connection refused"
Jul 16 22:11:20.287752 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.287722    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:11:20.330617 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.330579    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:11:20.431395 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.431350    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls\") pod \"36437163-7ef1-4122-93fc-808ae5ae10a1\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") "
Jul 16 22:11:20.431606 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.431431    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b29823-fd41-4147-959f-e09b96ad0567-kube-rbac-proxy-sar-config\") pod \"d6b29823-fd41-4147-959f-e09b96ad0567\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") "
Jul 16 22:11:20.431606 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.431455    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-rbac-proxy-sar-config\") pod \"36437163-7ef1-4122-93fc-808ae5ae10a1\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") "
Jul 16 22:11:20.431606 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.431503    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmc9q\" (UniqueName: \"kubernetes.io/projected/d6b29823-fd41-4147-959f-e09b96ad0567-kube-api-access-nmc9q\") pod \"d6b29823-fd41-4147-959f-e09b96ad0567\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") "
Jul 16 22:11:20.431786 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.431648    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b29823-fd41-4147-959f-e09b96ad0567-proxy-tls\") pod \"d6b29823-fd41-4147-959f-e09b96ad0567\" (UID: \"d6b29823-fd41-4147-959f-e09b96ad0567\") "
Jul 16 22:11:20.431786 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.431694    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8qwwr\" (UniqueName: \"kubernetes.io/projected/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-api-access-8qwwr\") pod \"36437163-7ef1-4122-93fc-808ae5ae10a1\" (UID: \"36437163-7ef1-4122-93fc-808ae5ae10a1\") "
Jul 16 22:11:20.431907 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.431868    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b29823-fd41-4147-959f-e09b96ad0567-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "d6b29823-fd41-4147-959f-e09b96ad0567" (UID: "d6b29823-fd41-4147-959f-e09b96ad0567"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:11:20.431907 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.431880    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "36437163-7ef1-4122-93fc-808ae5ae10a1" (UID: "36437163-7ef1-4122-93fc-808ae5ae10a1"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:11:20.432120 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.432080    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/d6b29823-fd41-4147-959f-e09b96ad0567-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:20.432190 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.432137    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:20.433841 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.433807    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "36437163-7ef1-4122-93fc-808ae5ae10a1" (UID: "36437163-7ef1-4122-93fc-808ae5ae10a1"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:11:20.433951 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.433847    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6b29823-fd41-4147-959f-e09b96ad0567-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "d6b29823-fd41-4147-959f-e09b96ad0567" (UID: "d6b29823-fd41-4147-959f-e09b96ad0567"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:11:20.434050 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.434020    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b29823-fd41-4147-959f-e09b96ad0567-kube-api-access-nmc9q" (OuterVolumeSpecName: "kube-api-access-nmc9q") pod "d6b29823-fd41-4147-959f-e09b96ad0567" (UID: "d6b29823-fd41-4147-959f-e09b96ad0567"). InnerVolumeSpecName "kube-api-access-nmc9q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:11:20.434085 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.434048    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-api-access-8qwwr" (OuterVolumeSpecName: "kube-api-access-8qwwr") pod "36437163-7ef1-4122-93fc-808ae5ae10a1" (UID: "36437163-7ef1-4122-93fc-808ae5ae10a1"). InnerVolumeSpecName "kube-api-access-8qwwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:11:20.500768 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.500733    2585 generic.go:356] "Generic (PLEG): container finished" podID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerID="e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085" exitCode=0
Jul 16 22:11:20.501232 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.500786    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"
Jul 16 22:11:20.501232 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.500816    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" event={"ID":"36437163-7ef1-4122-93fc-808ae5ae10a1","Type":"ContainerDied","Data":"e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085"}
Jul 16 22:11:20.501232 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.500875    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r" event={"ID":"36437163-7ef1-4122-93fc-808ae5ae10a1","Type":"ContainerDied","Data":"f3c0292380a3200f342ecdd1d64e66845d9b85ffa47542154fe24c869d95fee4"}
Jul 16 22:11:20.501232 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.500901    2585 scope.go:117] "RemoveContainer" containerID="f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b"
Jul 16 22:11:20.502798 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.502772    2585 generic.go:356] "Generic (PLEG): container finished" podID="d6b29823-fd41-4147-959f-e09b96ad0567" containerID="44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38" exitCode=0
Jul 16 22:11:20.502913 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.502806    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" event={"ID":"d6b29823-fd41-4147-959f-e09b96ad0567","Type":"ContainerDied","Data":"44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38"}
Jul 16 22:11:20.502913 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.502833    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"
Jul 16 22:11:20.502913 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.502852    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw" event={"ID":"d6b29823-fd41-4147-959f-e09b96ad0567","Type":"ContainerDied","Data":"fa6fcc05dfbddc58627c65800b413337317e9c181569c222d0159706ae577f20"}
Jul 16 22:11:20.514703 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.514673    2585 scope.go:117] "RemoveContainer" containerID="e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085"
Jul 16 22:11:20.524580 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.524553    2585 scope.go:117] "RemoveContainer" containerID="f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b"
Jul 16 22:11:20.524890 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:20.524870    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b\": container with ID starting with f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b not found: ID does not exist" containerID="f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b"
Jul 16 22:11:20.524983 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.524898    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b"} err="failed to get container status \"f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b\": rpc error: code = NotFound desc = could not find container \"f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b\": container with ID starting with f0ca4066bbbc9552a0b038afa8dbbf97e7d10b827c9db30a0cc5aa14bc1ca21b not found: ID does not exist"
Jul 16 22:11:20.524983 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.524917    2585 scope.go:117] "RemoveContainer" containerID="e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085"
Jul 16 22:11:20.525196 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:20.525154    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085\": container with ID starting with e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085 not found: ID does not exist" containerID="e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085"
Jul 16 22:11:20.525290 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.525200    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085"} err="failed to get container status \"e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085\": rpc error: code = NotFound desc = could not find container \"e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085\": container with ID starting with e686bda3760f0b0b5192d6336ee099c57973b21832b645dcfef110639fc96085 not found: ID does not exist"
Jul 16 22:11:20.525290 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.525218    2585 scope.go:117] "RemoveContainer" containerID="ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4"
Jul 16 22:11:20.533496 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.533471    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmc9q\" (UniqueName: \"kubernetes.io/projected/d6b29823-fd41-4147-959f-e09b96ad0567-kube-api-access-nmc9q\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:20.533496 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.533496    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/d6b29823-fd41-4147-959f-e09b96ad0567-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:20.533637 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.533507    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8qwwr\" (UniqueName: \"kubernetes.io/projected/36437163-7ef1-4122-93fc-808ae5ae10a1-kube-api-access-8qwwr\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:20.533637 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.533516    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/36437163-7ef1-4122-93fc-808ae5ae10a1-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:20.534202 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.534185    2585 scope.go:117] "RemoveContainer" containerID="44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38"
Jul 16 22:11:20.542591 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.542479    2585 scope.go:117] "RemoveContainer" containerID="ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4"
Jul 16 22:11:20.542845 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:20.542818    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4\": container with ID starting with ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4 not found: ID does not exist" containerID="ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4"
Jul 16 22:11:20.542962 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.542855    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4"} err="failed to get container status \"ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4\": rpc error: code = NotFound desc = could not find container \"ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4\": container with ID starting with ae7ea6d01f15e12f43790663aeeba57fd8d7cf91aceac806de3cee768d17ebd4 not found: ID does not exist"
Jul 16 22:11:20.542962 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.542876    2585 scope.go:117] "RemoveContainer" containerID="44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38"
Jul 16 22:11:20.543203 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:20.543186    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38\": container with ID starting with 44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38 not found: ID does not exist" containerID="44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38"
Jul 16 22:11:20.543265 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.543209    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38"} err="failed to get container status \"44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38\": rpc error: code = NotFound desc = could not find container \"44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38\": container with ID starting with 44bc5a0cd0c23be8b4051caf450307b626f14bd1188ee6364de4da1bb9774d38 not found: ID does not exist"
Jul 16 22:11:20.547582 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.547554    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"]
Jul 16 22:11:20.551875 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.551847    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-69517-predictor-86957765c5-22c7r"]
Jul 16 22:11:20.569004 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.568959    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"]
Jul 16 22:11:20.580641 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.580594    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-69517-predictor-85bc9bd758-gtdkw"]
Jul 16 22:11:20.828239 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.828151    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" path="/var/lib/kubelet/pods/36437163-7ef1-4122-93fc-808ae5ae10a1/volumes"
Jul 16 22:11:20.828627 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:20.828613    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" path="/var/lib/kubelet/pods/d6b29823-fd41-4147-959f-e09b96ad0567/volumes"
Jul 16 22:11:24.497367 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:24.497330    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:11:24.498045 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:24.498013    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 16 22:11:24.498572 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:24.498551    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:11:24.499056 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:24.499028    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 16 22:11:34.498730 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:34.498688    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 16 22:11:34.499153 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:34.499027    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 16 22:11:44.498762 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:44.498719    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 16 22:11:44.499185 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:44.499030    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 16 22:11:51.926892 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:51.926844    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:51.986473 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:51.986438    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"]
Jul 16 22:11:51.986739 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:51.986692    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" containerID="cri-o://235fda80d09acdf0d533f075db44068bba4e9e0b4837256161dc4f94d76b5a93" gracePeriod=30
Jul 16 22:11:51.986880 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:51.986764    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kube-rbac-proxy" containerID="cri-o://15ec9475679c90cdc8c33acda16615274d8533284db6197bfd71ff3f26a2c43a" gracePeriod=30
Jul 16 22:11:52.015761 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.015713    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:52.034708 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.034667    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"]
Jul 16 22:11:52.034931 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.034894    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" containerID="cri-o://bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50" gracePeriod=30
Jul 16 22:11:52.035068 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.034934    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kube-rbac-proxy" containerID="cri-o://f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50" gracePeriod=30
Jul 16 22:11:52.095274 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.095238    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:52.128422 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128376    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"]
Jul 16 22:11:52.128871 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128858    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container"
Jul 16 22:11:52.128921 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128873    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container"
Jul 16 22:11:52.128921 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128884    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kube-rbac-proxy"
Jul 16 22:11:52.128921 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128890    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kube-rbac-proxy"
Jul 16 22:11:52.128921 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128904    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container"
Jul 16 22:11:52.128921 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128911    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container"
Jul 16 22:11:52.128921 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128917    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kube-rbac-proxy"
Jul 16 22:11:52.128921 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128923    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kube-rbac-proxy"
Jul 16 22:11:52.129516 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.128998    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kserve-container"
Jul 16 22:11:52.129516 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.129004    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="36437163-7ef1-4122-93fc-808ae5ae10a1" containerName="kube-rbac-proxy"
Jul 16 22:11:52.129516 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.129014    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kserve-container"
Jul 16 22:11:52.129516 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.129019    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="d6b29823-fd41-4147-959f-e09b96ad0567" containerName="kube-rbac-proxy"
Jul 16 22:11:52.132724 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.132700    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.135350 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.135325    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-67446-predictor-serving-cert\""
Jul 16 22:11:52.135477 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.135375    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-67446-kube-rbac-proxy-sar-config\""
Jul 16 22:11:52.144151 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.144121    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"]
Jul 16 22:11:52.196341 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.196251    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:52.228546 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.228502    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"]
Jul 16 22:11:52.229404 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.229374    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.229542 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.229425    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f981066-32e7-46af-b68c-6a64857028b8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.229542 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.229520    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxrkm\" (UniqueName: \"kubernetes.io/projected/8f981066-32e7-46af-b68c-6a64857028b8-kube-api-access-gxrkm\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.232578 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.232547    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.234859 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.234834    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-67446-kube-rbac-proxy-sar-config\""
Jul 16 22:11:52.234973 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.234878    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-67446-predictor-serving-cert\""
Jul 16 22:11:52.246740 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.246708    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"]
Jul 16 22:11:52.271510 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.271461    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:52.330357 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.330320    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.330549 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.330380    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.330549 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.330411    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f981066-32e7-46af-b68c-6a64857028b8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.330549 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.330465    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9z4t\" (UniqueName: \"kubernetes.io/projected/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-api-access-b9z4t\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.330549 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:52.330485    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-67446-predictor-serving-cert: secret "success-200-isvc-67446-predictor-serving-cert" not found
Jul 16 22:11:52.330720 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:52.330555    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls podName:8f981066-32e7-46af-b68c-6a64857028b8 nodeName:}" failed. No retries permitted until 2026-07-16 22:11:52.830537833 +0000 UTC m=+1312.634961140 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls") pod "success-200-isvc-67446-predictor-58758849c6-zml4x" (UID: "8f981066-32e7-46af-b68c-6a64857028b8") : secret "success-200-isvc-67446-predictor-serving-cert" not found
Jul 16 22:11:52.330720 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.330606    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25fc5316-4d1e-480a-a6e4-db45339a8d09-proxy-tls\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.330720 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.330706    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxrkm\" (UniqueName: \"kubernetes.io/projected/8f981066-32e7-46af-b68c-6a64857028b8-kube-api-access-gxrkm\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.331163 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.331146    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f981066-32e7-46af-b68c-6a64857028b8-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.340985 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.340936    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:52.343309 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.343283    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxrkm\" (UniqueName: \"kubernetes.io/projected/8f981066-32e7-46af-b68c-6a64857028b8-kube-api-access-gxrkm\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.432082 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.432044    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.432311 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.432165    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9z4t\" (UniqueName: \"kubernetes.io/projected/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-api-access-b9z4t\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.432311 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.432198    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25fc5316-4d1e-480a-a6e4-db45339a8d09-proxy-tls\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.432891 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.432860    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.434748 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.434726    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25fc5316-4d1e-480a-a6e4-db45339a8d09-proxy-tls\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.444817 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.444789    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9z4t\" (UniqueName: \"kubernetes.io/projected/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-api-access-b9z4t\") pod \"error-404-isvc-67446-predictor-86c85bf98f-wwgzs\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") " pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.543277 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.543234    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:52.575175 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.575137    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:11:52.679647 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.679610    2585 generic.go:356] "Generic (PLEG): container finished" podID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerID="f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50" exitCode=2
Jul 16 22:11:52.679834 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.679679    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" event={"ID":"8832ac4b-a3b2-4a46-82ca-c9049562d2e7","Type":"ContainerDied","Data":"f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50"}
Jul 16 22:11:52.681617 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.681588    2585 generic.go:356] "Generic (PLEG): container finished" podID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerID="15ec9475679c90cdc8c33acda16615274d8533284db6197bfd71ff3f26a2c43a" exitCode=2
Jul 16 22:11:52.681756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.681656    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" event={"ID":"b2a4d504-f247-461b-9f65-b8a83e2837d6","Type":"ContainerDied","Data":"15ec9475679c90cdc8c33acda16615274d8533284db6197bfd71ff3f26a2c43a"}
Jul 16 22:11:52.692487 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.692458    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"]
Jul 16 22:11:52.694437 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:11:52.694407    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25fc5316_4d1e_480a_a6e4_db45339a8d09.slice/crio-a4e2e4a83ba1b2152f27876f1d10d61675a6764f4df0cdc9e5699b1dedb98df4 WatchSource:0}: Error finding container a4e2e4a83ba1b2152f27876f1d10d61675a6764f4df0cdc9e5699b1dedb98df4: Status 404 returned error can't find the container with id a4e2e4a83ba1b2152f27876f1d10d61675a6764f4df0cdc9e5699b1dedb98df4
Jul 16 22:11:52.836791 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.836763    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:52.839324 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:52.839297    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls\") pod \"success-200-isvc-67446-predictor-58758849c6-zml4x\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") " pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:53.045143 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.045081    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:53.192021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.191992    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"]
Jul 16 22:11:53.194147 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:11:53.194117    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f981066_32e7_46af_b68c_6a64857028b8.slice/crio-9d230b27ba402aaf2c378e8f23a3fe50ccce49e6b6eb1d54f6517f26024b0b3b WatchSource:0}: Error finding container 9d230b27ba402aaf2c378e8f23a3fe50ccce49e6b6eb1d54f6517f26024b0b3b: Status 404 returned error can't find the container with id 9d230b27ba402aaf2c378e8f23a3fe50ccce49e6b6eb1d54f6517f26024b0b3b
Jul 16 22:11:53.689201 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.689157    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" event={"ID":"8f981066-32e7-46af-b68c-6a64857028b8","Type":"ContainerStarted","Data":"6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99"}
Jul 16 22:11:53.689409 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.689330    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:53.689409 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.689383    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" event={"ID":"8f981066-32e7-46af-b68c-6a64857028b8","Type":"ContainerStarted","Data":"d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b"}
Jul 16 22:11:53.689518 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.689415    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" event={"ID":"8f981066-32e7-46af-b68c-6a64857028b8","Type":"ContainerStarted","Data":"9d230b27ba402aaf2c378e8f23a3fe50ccce49e6b6eb1d54f6517f26024b0b3b"}
Jul 16 22:11:53.690988 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.690962    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" event={"ID":"25fc5316-4d1e-480a-a6e4-db45339a8d09","Type":"ContainerStarted","Data":"fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0"}
Jul 16 22:11:53.691135 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.691002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" event={"ID":"25fc5316-4d1e-480a-a6e4-db45339a8d09","Type":"ContainerStarted","Data":"df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4"}
Jul 16 22:11:53.691135 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.691031    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" event={"ID":"25fc5316-4d1e-480a-a6e4-db45339a8d09","Type":"ContainerStarted","Data":"a4e2e4a83ba1b2152f27876f1d10d61675a6764f4df0cdc9e5699b1dedb98df4"}
Jul 16 22:11:53.691135 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.691063    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:53.710510 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.710442    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podStartSLOduration=1.710389363 podStartE2EDuration="1.710389363s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:11:53.706805841 +0000 UTC m=+1313.511229167" watchObservedRunningTime="2026-07-16 22:11:53.710389363 +0000 UTC m=+1313.514812689"
Jul 16 22:11:53.726931 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:53.726860    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podStartSLOduration=1.726841588 podStartE2EDuration="1.726841588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:11:53.725157613 +0000 UTC m=+1313.529580940" watchObservedRunningTime="2026-07-16 22:11:53.726841588 +0000 UTC m=+1313.531265005"
Jul 16 22:11:54.498581 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:54.498541    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 16 22:11:54.499033 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:54.499006    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 16 22:11:54.696974 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:54.696929    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:11:54.697208 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:54.696999    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:11:54.697928 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:54.697896    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 16 22:11:54.697928 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:54.697909    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 16 22:11:54.824955 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:54.824858    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.44:8643/healthz\": dial tcp 10.134.0.44:8643: connect: connection refused"
Jul 16 22:11:54.830174 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:54.830143    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.44:8080: connect: connection refused"
Jul 16 22:11:55.702379 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:55.702333    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 16 22:11:55.702790 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:55.702395    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 16 22:11:55.829795 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:55.829746    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.43:8643/healthz\": dial tcp 10.134.0.43:8643: connect: connection refused"
Jul 16 22:11:55.834408 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:55.834373    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.43:8080: connect: connection refused"
Jul 16 22:11:56.094904 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.094874    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:11:56.168832 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.168789    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-rbac-proxy-sar-config\") pod \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") "
Jul 16 22:11:56.169010 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.168856    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-proxy-tls\") pod \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") "
Jul 16 22:11:56.169010 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.168887    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-md4gc\" (UniqueName: \"kubernetes.io/projected/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-api-access-md4gc\") pod \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\" (UID: \"8832ac4b-a3b2-4a46-82ca-c9049562d2e7\") "
Jul 16 22:11:56.169349 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.169308    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8832ac4b-a3b2-4a46-82ca-c9049562d2e7" (UID: "8832ac4b-a3b2-4a46-82ca-c9049562d2e7"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:11:56.171024 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.170995    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-api-access-md4gc" (OuterVolumeSpecName: "kube-api-access-md4gc") pod "8832ac4b-a3b2-4a46-82ca-c9049562d2e7" (UID: "8832ac4b-a3b2-4a46-82ca-c9049562d2e7"). InnerVolumeSpecName "kube-api-access-md4gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:11:56.171140 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.171050    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8832ac4b-a3b2-4a46-82ca-c9049562d2e7" (UID: "8832ac4b-a3b2-4a46-82ca-c9049562d2e7"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:11:56.270346 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.270250    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:56.270346 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.270284    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:56.270346 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.270296    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-md4gc\" (UniqueName: \"kubernetes.io/projected/8832ac4b-a3b2-4a46-82ca-c9049562d2e7-kube-api-access-md4gc\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:56.710466 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.710412    2585 generic.go:356] "Generic (PLEG): container finished" podID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerID="bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50" exitCode=0
Jul 16 22:11:56.710954 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.710482    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"
Jul 16 22:11:56.710954 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.710534    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" event={"ID":"8832ac4b-a3b2-4a46-82ca-c9049562d2e7","Type":"ContainerDied","Data":"bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50"}
Jul 16 22:11:56.710954 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.710632    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd" event={"ID":"8832ac4b-a3b2-4a46-82ca-c9049562d2e7","Type":"ContainerDied","Data":"a129622b4fbcceb766f0de20c232772f5d864ed91a3525f331d918984d3e0f53"}
Jul 16 22:11:56.710954 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.710671    2585 scope.go:117] "RemoveContainer" containerID="f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50"
Jul 16 22:11:56.714432 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.714152    2585 generic.go:356] "Generic (PLEG): container finished" podID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerID="235fda80d09acdf0d533f075db44068bba4e9e0b4837256161dc4f94d76b5a93" exitCode=0
Jul 16 22:11:56.714432 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.714272    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" event={"ID":"b2a4d504-f247-461b-9f65-b8a83e2837d6","Type":"ContainerDied","Data":"235fda80d09acdf0d533f075db44068bba4e9e0b4837256161dc4f94d76b5a93"}
Jul 16 22:11:56.723788 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.723761    2585 scope.go:117] "RemoveContainer" containerID="bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50"
Jul 16 22:11:56.733347 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.733314    2585 scope.go:117] "RemoveContainer" containerID="f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50"
Jul 16 22:11:56.733867 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:56.733825    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50\": container with ID starting with f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50 not found: ID does not exist" containerID="f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50"
Jul 16 22:11:56.733986 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.733888    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50"} err="failed to get container status \"f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50\": rpc error: code = NotFound desc = could not find container \"f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50\": container with ID starting with f009bd6782def73039a8cc9efb828a4617c85f06baafbe5ef30b5ab338667d50 not found: ID does not exist"
Jul 16 22:11:56.733986 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.733919    2585 scope.go:117] "RemoveContainer" containerID="bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50"
Jul 16 22:11:56.734486 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:11:56.734463    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50\": container with ID starting with bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50 not found: ID does not exist" containerID="bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50"
Jul 16 22:11:56.734628 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.734493    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50"} err="failed to get container status \"bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50\": rpc error: code = NotFound desc = could not find container \"bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50\": container with ID starting with bb926065cc11ee316829f21b60b69fa9981e44144bd4a599dbb47000e560aa50 not found: ID does not exist"
Jul 16 22:11:56.736644 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.736620    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"]
Jul 16 22:11:56.740524 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.740498    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-77f35-predictor-766f77d998-qfjbd"]
Jul 16 22:11:56.744729 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.744704    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:11:56.827472 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.827436    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" path="/var/lib/kubelet/pods/8832ac4b-a3b2-4a46-82ca-c9049562d2e7/volumes"
Jul 16 22:11:56.876666 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.876567    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sqqm2\" (UniqueName: \"kubernetes.io/projected/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-api-access-sqqm2\") pod \"b2a4d504-f247-461b-9f65-b8a83e2837d6\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") "
Jul 16 22:11:56.876666 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.876627    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-rbac-proxy-sar-config\") pod \"b2a4d504-f247-461b-9f65-b8a83e2837d6\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") "
Jul 16 22:11:56.876666 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.876668    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls\") pod \"b2a4d504-f247-461b-9f65-b8a83e2837d6\" (UID: \"b2a4d504-f247-461b-9f65-b8a83e2837d6\") "
Jul 16 22:11:56.877068 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.877043    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "b2a4d504-f247-461b-9f65-b8a83e2837d6" (UID: "b2a4d504-f247-461b-9f65-b8a83e2837d6"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:11:56.878871 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.878844    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "b2a4d504-f247-461b-9f65-b8a83e2837d6" (UID: "b2a4d504-f247-461b-9f65-b8a83e2837d6"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:11:56.878972 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.878899    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-api-access-sqqm2" (OuterVolumeSpecName: "kube-api-access-sqqm2") pod "b2a4d504-f247-461b-9f65-b8a83e2837d6" (UID: "b2a4d504-f247-461b-9f65-b8a83e2837d6"). InnerVolumeSpecName "kube-api-access-sqqm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:11:56.978267 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.978232    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/b2a4d504-f247-461b-9f65-b8a83e2837d6-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:56.978267 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.978265    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqqm2\" (UniqueName: \"kubernetes.io/projected/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-api-access-sqqm2\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:56.978267 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:56.978275    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/b2a4d504-f247-461b-9f65-b8a83e2837d6-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:11:57.723182 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:57.723156    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"
Jul 16 22:11:57.723182 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:57.723165    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z" event={"ID":"b2a4d504-f247-461b-9f65-b8a83e2837d6","Type":"ContainerDied","Data":"f0b5595fafea8d38ab59dd61f25569449609c0e658506cb08280c5e3df432611"}
Jul 16 22:11:57.723647 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:57.723237    2585 scope.go:117] "RemoveContainer" containerID="15ec9475679c90cdc8c33acda16615274d8533284db6197bfd71ff3f26a2c43a"
Jul 16 22:11:57.732873 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:57.732854    2585 scope.go:117] "RemoveContainer" containerID="235fda80d09acdf0d533f075db44068bba4e9e0b4837256161dc4f94d76b5a93"
Jul 16 22:11:57.746855 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:57.746824    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"]
Jul 16 22:11:57.748774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:57.748750    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-77f35-predictor-558485f7c5-gvp7z"]
Jul 16 22:11:58.827153 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:11:58.827119    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" path="/var/lib/kubelet/pods/b2a4d504-f247-461b-9f65-b8a83e2837d6/volumes"
Jul 16 22:12:00.706597 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:00.706559    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:12:00.706987 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:00.706649    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:12:00.707134 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:00.707085    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 16 22:12:00.707318 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:00.707293    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 16 22:12:04.498179 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:04.498128    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.46:8080: connect: connection refused"
Jul 16 22:12:04.499569 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:04.499524    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.45:8080: connect: connection refused"
Jul 16 22:12:10.708040 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:10.707993    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 16 22:12:10.708565 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:10.707991    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 16 22:12:14.499168 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:14.499137    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:12:14.500165 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:14.500148    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:12:20.707252 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:20.707197    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 16 22:12:20.707705 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:20.707289    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 16 22:12:26.247599 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:26.247550    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:12:26.409590 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:26.409525    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:12:30.707301 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:30.707262    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 16 22:12:30.707697 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:30.707262    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 16 22:12:36.343950 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.343900    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:12:36.474117 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.474057    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"]
Jul 16 22:12:36.475011 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.474971    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kube-rbac-proxy" containerID="cri-o://6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a" gracePeriod=30
Jul 16 22:12:36.477086 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.476919    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container" containerID="cri-o://031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e" gracePeriod=30
Jul 16 22:12:36.486403 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.486371    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:12:36.528437 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.528401    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"]
Jul 16 22:12:36.529248 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529181    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kube-rbac-proxy"
Jul 16 22:12:36.529248 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529211    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kube-rbac-proxy"
Jul 16 22:12:36.529472 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529250    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container"
Jul 16 22:12:36.529472 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529275    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container"
Jul 16 22:12:36.529472 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529301    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kube-rbac-proxy"
Jul 16 22:12:36.529472 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529325    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kube-rbac-proxy"
Jul 16 22:12:36.529472 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529347    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container"
Jul 16 22:12:36.529472 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529357    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container"
Jul 16 22:12:36.529756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529487    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kserve-container"
Jul 16 22:12:36.529756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529505    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kserve-container"
Jul 16 22:12:36.529756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529516    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2a4d504-f247-461b-9f65-b8a83e2837d6" containerName="kube-rbac-proxy"
Jul 16 22:12:36.529756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.529525    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8832ac4b-a3b2-4a46-82ca-c9049562d2e7" containerName="kube-rbac-proxy"
Jul 16 22:12:36.532943 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.532918    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.535633 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.535604    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c1850-predictor-serving-cert\""
Jul 16 22:12:36.535776 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.535643    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-c1850-kube-rbac-proxy-sar-config\""
Jul 16 22:12:36.537371 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.537316    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"]
Jul 16 22:12:36.537681 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.537626    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container" containerID="cri-o://dc6f23338d1f18c00e1571c92bbeb87b4f0ea934e0e350228ca1e2d3e53198f7" gracePeriod=30
Jul 16 22:12:36.538167 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.537692    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kube-rbac-proxy" containerID="cri-o://c57afc52fa17aea0b267d35431d36c81a676a7f01bde9afe928216c9b0f18606" gracePeriod=30
Jul 16 22:12:36.545602 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.545571    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"]
Jul 16 22:12:36.588977 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.588943    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:12:36.653973 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.653943    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"]
Jul 16 22:12:36.657604 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.657585    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.660179 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.660159    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c1850-predictor-serving-cert\""
Jul 16 22:12:36.660277 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.660226    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-c1850-kube-rbac-proxy-sar-config\""
Jul 16 22:12:36.671843 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.671804    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"]
Jul 16 22:12:36.671983 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.671872    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56149faf-9310-48bd-b667-2f9e9216898e-proxy-tls\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.671983 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.671968    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56149faf-9310-48bd-b667-2f9e9216898e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.672071 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.672046    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q774\" (UniqueName: \"kubernetes.io/projected/56149faf-9310-48bd-b667-2f9e9216898e-kube-api-access-7q774\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.694781 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.694740    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:12:36.773185 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.773142    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a47779-330b-4d20-87ba-123ae324eea8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.773372 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.773256    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56149faf-9310-48bd-b667-2f9e9216898e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.773372 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.773295    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a47779-330b-4d20-87ba-123ae324eea8-proxy-tls\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.773372 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.773322    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5525h\" (UniqueName: \"kubernetes.io/projected/86a47779-330b-4d20-87ba-123ae324eea8-kube-api-access-5525h\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.773372 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.773360    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q774\" (UniqueName: \"kubernetes.io/projected/56149faf-9310-48bd-b667-2f9e9216898e-kube-api-access-7q774\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.773571 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.773513    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56149faf-9310-48bd-b667-2f9e9216898e-proxy-tls\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.773913 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.773885    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56149faf-9310-48bd-b667-2f9e9216898e-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.776301 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.776275    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56149faf-9310-48bd-b667-2f9e9216898e-proxy-tls\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.783328 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.783298    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:12:36.783527 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.783505    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q774\" (UniqueName: \"kubernetes.io/projected/56149faf-9310-48bd-b667-2f9e9216898e-kube-api-access-7q774\") pod \"success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") " pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.849431 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.849389    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:36.874743 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.874708    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a47779-330b-4d20-87ba-123ae324eea8-proxy-tls\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.874914 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.874767    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5525h\" (UniqueName: \"kubernetes.io/projected/86a47779-330b-4d20-87ba-123ae324eea8-kube-api-access-5525h\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.874914 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.874873    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a47779-330b-4d20-87ba-123ae324eea8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.875678 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.875655    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a47779-330b-4d20-87ba-123ae324eea8-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.877607 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.877580    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a47779-330b-4d20-87ba-123ae324eea8-proxy-tls\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.884698 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.884667    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5525h\" (UniqueName: \"kubernetes.io/projected/86a47779-330b-4d20-87ba-123ae324eea8-kube-api-access-5525h\") pod \"error-404-isvc-c1850-predictor-597c44d6bf-wwcv5\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") " pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:36.925623 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.925589    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:12:36.938803 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.938756    2585 generic.go:356] "Generic (PLEG): container finished" podID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerID="c57afc52fa17aea0b267d35431d36c81a676a7f01bde9afe928216c9b0f18606" exitCode=2
Jul 16 22:12:36.938803 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.938793    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" event={"ID":"6ce564fd-bfcf-4e1b-8430-7347fb64adaa","Type":"ContainerDied","Data":"c57afc52fa17aea0b267d35431d36c81a676a7f01bde9afe928216c9b0f18606"}
Jul 16 22:12:36.940877 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.940843    2585 generic.go:356] "Generic (PLEG): container finished" podID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerID="6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a" exitCode=2
Jul 16 22:12:36.941010 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.940882    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" event={"ID":"890b57c2-207d-4f3b-ba55-5607db805fd0","Type":"ContainerDied","Data":"6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a"}
Jul 16 22:12:36.969476 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:36.969436    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:37.012958 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.011415    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"]
Jul 16 22:12:37.133672 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.133592    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"]
Jul 16 22:12:37.136395 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:12:37.136366    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86a47779_330b_4d20_87ba_123ae324eea8.slice/crio-d0f1e47e9deca5352967411d27cff8ab47306ba5204c2bcd7adb0dbf5a683ec4 WatchSource:0}: Error finding container d0f1e47e9deca5352967411d27cff8ab47306ba5204c2bcd7adb0dbf5a683ec4: Status 404 returned error can't find the container with id d0f1e47e9deca5352967411d27cff8ab47306ba5204c2bcd7adb0dbf5a683ec4
Jul 16 22:12:37.948310 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.948270    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" event={"ID":"86a47779-330b-4d20-87ba-123ae324eea8","Type":"ContainerStarted","Data":"d91b7016082b787872f46e81f857660dc971f06baf8c0f7955d234a72e04628b"}
Jul 16 22:12:37.948770 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.948338    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" event={"ID":"86a47779-330b-4d20-87ba-123ae324eea8","Type":"ContainerStarted","Data":"11214c5bebf4a27d07356dde35e626614d40e1c8cc77b0bb5920b1d04f98ea04"}
Jul 16 22:12:37.948770 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.948365    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" event={"ID":"86a47779-330b-4d20-87ba-123ae324eea8","Type":"ContainerStarted","Data":"d0f1e47e9deca5352967411d27cff8ab47306ba5204c2bcd7adb0dbf5a683ec4"}
Jul 16 22:12:37.948770 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.948513    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:37.948770 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.948575    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:37.950145 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.950110    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 16 22:12:37.950405 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.950379    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" event={"ID":"56149faf-9310-48bd-b667-2f9e9216898e","Type":"ContainerStarted","Data":"a1d0601393e8e135ef9f9768bd059b0b02f91ad1512c856b74996117c8c999c3"}
Jul 16 22:12:37.950464 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.950441    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" event={"ID":"56149faf-9310-48bd-b667-2f9e9216898e","Type":"ContainerStarted","Data":"bec91a44e26a78cccf0185467b9bc44589c87146c1992c98fe8ceda7ec271242"}
Jul 16 22:12:37.950499 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.950476    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:37.950499 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:37.950494    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" event={"ID":"56149faf-9310-48bd-b667-2f9e9216898e","Type":"ContainerStarted","Data":"f8f06c3abdffa6f6d9b27b5f946a11bf8f4afc6e40385b41d2f93d1765f4927e"}
Jul 16 22:12:38.005118 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:38.005028    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podStartSLOduration=2.005000124 podStartE2EDuration="2.005000124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:12:37.977119217 +0000 UTC m=+1357.781542544" watchObservedRunningTime="2026-07-16 22:12:38.005000124 +0000 UTC m=+1357.809423450"
Jul 16 22:12:38.956420 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:38.956384    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:38.956836 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:38.956502    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 16 22:12:38.957946 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:38.957915    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 16 22:12:39.493180 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:39.493094    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.45:8643/healthz\": dial tcp 10.134.0.45:8643: connect: connection refused"
Jul 16 22:12:39.493180 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:39.493173    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.46:8643/healthz\": dial tcp 10.134.0.46:8643: connect: connection refused"
Jul 16 22:12:39.962393 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:39.962291    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 16 22:12:40.707232 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.707186    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.48:8080: connect: connection refused"
Jul 16 22:12:40.707392 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.707370    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.47:8080: connect: connection refused"
Jul 16 22:12:40.851307 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.851280    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:12:40.872862 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.872755    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podStartSLOduration=4.872732605 podStartE2EDuration="4.872732605s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:12:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:12:38.006698618 +0000 UTC m=+1357.811121944" watchObservedRunningTime="2026-07-16 22:12:40.872732605 +0000 UTC m=+1360.677155931"
Jul 16 22:12:40.918191 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.917990    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ww5c8\" (UniqueName: \"kubernetes.io/projected/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-api-access-ww5c8\") pod \"890b57c2-207d-4f3b-ba55-5607db805fd0\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") "
Jul 16 22:12:40.918191 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.918151    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls\") pod \"890b57c2-207d-4f3b-ba55-5607db805fd0\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") "
Jul 16 22:12:40.918191 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.918181    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-rbac-proxy-sar-config\") pod \"890b57c2-207d-4f3b-ba55-5607db805fd0\" (UID: \"890b57c2-207d-4f3b-ba55-5607db805fd0\") "
Jul 16 22:12:40.918683 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.918658    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "890b57c2-207d-4f3b-ba55-5607db805fd0" (UID: "890b57c2-207d-4f3b-ba55-5607db805fd0"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:40.920529 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.920496    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-api-access-ww5c8" (OuterVolumeSpecName: "kube-api-access-ww5c8") pod "890b57c2-207d-4f3b-ba55-5607db805fd0" (UID: "890b57c2-207d-4f3b-ba55-5607db805fd0"). InnerVolumeSpecName "kube-api-access-ww5c8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:12:40.920665 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.920643    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "890b57c2-207d-4f3b-ba55-5607db805fd0" (UID: "890b57c2-207d-4f3b-ba55-5607db805fd0"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:12:40.969798 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.969758    2585 generic.go:356] "Generic (PLEG): container finished" podID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerID="031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e" exitCode=0
Jul 16 22:12:40.970362 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.969884    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"
Jul 16 22:12:40.970362 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.969885    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" event={"ID":"890b57c2-207d-4f3b-ba55-5607db805fd0","Type":"ContainerDied","Data":"031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e"}
Jul 16 22:12:40.970362 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.969975    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg" event={"ID":"890b57c2-207d-4f3b-ba55-5607db805fd0","Type":"ContainerDied","Data":"a5f09b40236e1a7992f122724c8d7c9ba61dc46416b6966c7ab81be0113e2232"}
Jul 16 22:12:40.970362 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.970011    2585 scope.go:117] "RemoveContainer" containerID="6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a"
Jul 16 22:12:40.972249 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.972223    2585 generic.go:356] "Generic (PLEG): container finished" podID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerID="dc6f23338d1f18c00e1571c92bbeb87b4f0ea934e0e350228ca1e2d3e53198f7" exitCode=0
Jul 16 22:12:40.972385 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.972290    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" event={"ID":"6ce564fd-bfcf-4e1b-8430-7347fb64adaa","Type":"ContainerDied","Data":"dc6f23338d1f18c00e1571c92bbeb87b4f0ea934e0e350228ca1e2d3e53198f7"}
Jul 16 22:12:40.983941 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.983917    2585 scope.go:117] "RemoveContainer" containerID="031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e"
Jul 16 22:12:40.994180 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.994151    2585 scope.go:117] "RemoveContainer" containerID="6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a"
Jul 16 22:12:40.994329 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.994276    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"]
Jul 16 22:12:40.994502 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:12:40.994482    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a\": container with ID starting with 6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a not found: ID does not exist" containerID="6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a"
Jul 16 22:12:40.994570 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.994511    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a"} err="failed to get container status \"6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a\": rpc error: code = NotFound desc = could not find container \"6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a\": container with ID starting with 6e41a8af32bbcf0b520d14765709ddceb0dca8e5c5202ba0e0ec8f77b602493a not found: ID does not exist"
Jul 16 22:12:40.994570 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.994540    2585 scope.go:117] "RemoveContainer" containerID="031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e"
Jul 16 22:12:40.994893 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:12:40.994873    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e\": container with ID starting with 031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e not found: ID does not exist" containerID="031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e"
Jul 16 22:12:40.994971 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.994897    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e"} err="failed to get container status \"031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e\": rpc error: code = NotFound desc = could not find container \"031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e\": container with ID starting with 031533e59747ea87c1708632bf2a25429575879ee15ec74bcfbc666b2467872e not found: ID does not exist"
Jul 16 22:12:40.997928 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:40.997898    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-1ab93-predictor-d659cd46b-gcpdg"]
Jul 16 22:12:41.019651 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.019620    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ww5c8\" (UniqueName: \"kubernetes.io/projected/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-api-access-ww5c8\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:12:41.019651 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.019651    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/890b57c2-207d-4f3b-ba55-5607db805fd0-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:12:41.019837 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.019661    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/890b57c2-207d-4f3b-ba55-5607db805fd0-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:12:41.097821 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.097788    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:12:41.221277 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.221168    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsl8b\" (UniqueName: \"kubernetes.io/projected/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-api-access-nsl8b\") pod \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") "
Jul 16 22:12:41.221277 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.221227    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-rbac-proxy-sar-config\") pod \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") "
Jul 16 22:12:41.221277 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.221257    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-proxy-tls\") pod \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\" (UID: \"6ce564fd-bfcf-4e1b-8430-7347fb64adaa\") "
Jul 16 22:12:41.221681 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.221643    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6ce564fd-bfcf-4e1b-8430-7347fb64adaa" (UID: "6ce564fd-bfcf-4e1b-8430-7347fb64adaa"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:12:41.223470 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.223442    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6ce564fd-bfcf-4e1b-8430-7347fb64adaa" (UID: "6ce564fd-bfcf-4e1b-8430-7347fb64adaa"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:12:41.223576 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.223472    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-api-access-nsl8b" (OuterVolumeSpecName: "kube-api-access-nsl8b") pod "6ce564fd-bfcf-4e1b-8430-7347fb64adaa" (UID: "6ce564fd-bfcf-4e1b-8430-7347fb64adaa"). InnerVolumeSpecName "kube-api-access-nsl8b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:12:41.322946 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.322900    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsl8b\" (UniqueName: \"kubernetes.io/projected/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-api-access-nsl8b\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:12:41.322946 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.322939    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:12:41.323180 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.322974    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6ce564fd-bfcf-4e1b-8430-7347fb64adaa-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:12:41.980119 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.980010    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"
Jul 16 22:12:41.980119 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.980037    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd" event={"ID":"6ce564fd-bfcf-4e1b-8430-7347fb64adaa","Type":"ContainerDied","Data":"ade0638d02f8cb3a794bc95e1058518e023fbe1e3424fe443fa56bdd96a84b2f"}
Jul 16 22:12:41.980651 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.980147    2585 scope.go:117] "RemoveContainer" containerID="c57afc52fa17aea0b267d35431d36c81a676a7f01bde9afe928216c9b0f18606"
Jul 16 22:12:41.989522 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:41.989501    2585 scope.go:117] "RemoveContainer" containerID="dc6f23338d1f18c00e1571c92bbeb87b4f0ea934e0e350228ca1e2d3e53198f7"
Jul 16 22:12:42.002246 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:42.002220    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"]
Jul 16 22:12:42.006486 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:42.006462    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-1ab93-predictor-7cd97fc847-qltmd"]
Jul 16 22:12:42.828077 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:42.828037    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" path="/var/lib/kubelet/pods/6ce564fd-bfcf-4e1b-8430-7347fb64adaa/volumes"
Jul 16 22:12:42.828562 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:42.828547    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" path="/var/lib/kubelet/pods/890b57c2-207d-4f3b-ba55-5607db805fd0/volumes"
Jul 16 22:12:43.961666 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:43.961636    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:12:43.962203 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:43.962177    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 16 22:12:44.966903 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:44.966875    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:12:44.967458 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:44.967430    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 16 22:12:50.708302 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:50.708267    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:12:50.708728 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:50.708403    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:12:53.962907 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:53.962857    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 16 22:12:54.967513 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:12:54.967473    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 16 22:13:02.150619 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:02.150582    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:02.409566 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:02.409468    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:03.963144 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:03.963073    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 16 22:13:04.968092 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:04.968043    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 16 22:13:12.271349 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.271317    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:12.345318 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.345272    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"]
Jul 16 22:13:12.345600 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.345543    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container" containerID="cri-o://df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4" gracePeriod=30
Jul 16 22:13:12.345758 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.345618    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kube-rbac-proxy" containerID="cri-o://fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0" gracePeriod=30
Jul 16 22:13:12.384545 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.384494    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:12.420197 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.420120    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"]
Jul 16 22:13:12.420467 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.420431    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container" containerID="cri-o://d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b" gracePeriod=30
Jul 16 22:13:12.420597 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.420445    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kube-rbac-proxy" containerID="cri-o://6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99" gracePeriod=30
Jul 16 22:13:12.479283 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479247    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"]
Jul 16 22:13:12.479714 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479699    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container"
Jul 16 22:13:12.479761 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479717    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container"
Jul 16 22:13:12.479761 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479729    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kube-rbac-proxy"
Jul 16 22:13:12.479761 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479734    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kube-rbac-proxy"
Jul 16 22:13:12.479761 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479756    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container"
Jul 16 22:13:12.479761 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479761    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container"
Jul 16 22:13:12.479911 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479782    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kube-rbac-proxy"
Jul 16 22:13:12.479911 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479787    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kube-rbac-proxy"
Jul 16 22:13:12.479911 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479843    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kube-rbac-proxy"
Jul 16 22:13:12.479911 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479849    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kube-rbac-proxy"
Jul 16 22:13:12.479911 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479857    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ce564fd-bfcf-4e1b-8430-7347fb64adaa" containerName="kserve-container"
Jul 16 22:13:12.479911 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.479865    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="890b57c2-207d-4f3b-ba55-5607db805fd0" containerName="kserve-container"
Jul 16 22:13:12.482159 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.482126    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:12.483372 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.483350    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.486731 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.486708    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a3a91-kube-rbac-proxy-sar-config\""
Jul 16 22:13:12.486861 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.486739    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a3a91-predictor-serving-cert\""
Jul 16 22:13:12.493904 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.493877    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"]
Jul 16 22:13:12.520581 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.520544    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.520581 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.520591    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.520837 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.520659    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qm2zd\" (UniqueName: \"kubernetes.io/projected/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-api-access-qm2zd\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.598611 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.598556    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:12.606752 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.606714    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"]
Jul 16 22:13:12.619664 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.619618    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:12.621473 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.621435    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.621618 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.621483    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.621618 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.621549    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qm2zd\" (UniqueName: \"kubernetes.io/projected/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-api-access-qm2zd\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.621618 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:12.621597    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-serving-cert: secret "success-200-isvc-a3a91-predictor-serving-cert" not found
Jul 16 22:13:12.621785 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:12.621670    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls podName:6de4c344-fd98-4d3d-a0d8-04cdd26a394a nodeName:}" failed. No retries permitted until 2026-07-16 22:13:13.121646051 +0000 UTC m=+1392.926069358 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls") pod "success-200-isvc-a3a91-predictor-d4f677b68-tggk7" (UID: "6de4c344-fd98-4d3d-a0d8-04cdd26a394a") : secret "success-200-isvc-a3a91-predictor-serving-cert" not found
Jul 16 22:13:12.622393 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.622367    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3a91-predictor-serving-cert\""
Jul 16 22:13:12.622747 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.622639    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3a91-kube-rbac-proxy-sar-config\""
Jul 16 22:13:12.622747 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.622674    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"]
Jul 16 22:13:12.622920 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.622760    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.631676 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.631646    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qm2zd\" (UniqueName: \"kubernetes.io/projected/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-api-access-qm2zd\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:12.722825 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.722776    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:12.723028 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.722922    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:12.723028 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.722989    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fz2kt\" (UniqueName: \"kubernetes.io/projected/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-api-access-fz2kt\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:12.824611 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.824524    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:12.824611 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.824572    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fz2kt\" (UniqueName: \"kubernetes.io/projected/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-api-access-fz2kt\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:12.824819 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.824643    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:12.824819 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:12.824783    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-serving-cert: secret "error-404-isvc-a3a91-predictor-serving-cert" not found
Jul 16 22:13:12.824911 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:12.824857    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls podName:6591ac58-f4a1-4d2b-ab1e-653071b8b3cd nodeName:}" failed. No retries permitted until 2026-07-16 22:13:13.324834672 +0000 UTC m=+1393.129257976 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls") pod "error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" (UID: "6591ac58-f4a1-4d2b-ab1e-653071b8b3cd") : secret "error-404-isvc-a3a91-predictor-serving-cert" not found
Jul 16 22:13:12.825404 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.825378    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:12.836231 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:12.836201    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fz2kt\" (UniqueName: \"kubernetes.io/projected/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-api-access-fz2kt\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:13.053393 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.053357    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:13.127982 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.127877    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:13.130438 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.130408    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls\") pod \"success-200-isvc-a3a91-predictor-d4f677b68-tggk7\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") " pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:13.150840 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.150800    2585 generic.go:356] "Generic (PLEG): container finished" podID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerID="fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0" exitCode=2
Jul 16 22:13:13.151014 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.150875    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" event={"ID":"25fc5316-4d1e-480a-a6e4-db45339a8d09","Type":"ContainerDied","Data":"fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0"}
Jul 16 22:13:13.156244 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.156210    2585 generic.go:356] "Generic (PLEG): container finished" podID="8f981066-32e7-46af-b68c-6a64857028b8" containerID="6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99" exitCode=2
Jul 16 22:13:13.156405 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.156269    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" event={"ID":"8f981066-32e7-46af-b68c-6a64857028b8","Type":"ContainerDied","Data":"6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99"}
Jul 16 22:13:13.330420 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.330378    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:13.330815 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:13.330546    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-serving-cert: secret "error-404-isvc-a3a91-predictor-serving-cert" not found
Jul 16 22:13:13.330815 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:13.330630    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls podName:6591ac58-f4a1-4d2b-ab1e-653071b8b3cd nodeName:}" failed. No retries permitted until 2026-07-16 22:13:14.33061442 +0000 UTC m=+1394.135037723 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls") pod "error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" (UID: "6591ac58-f4a1-4d2b-ab1e-653071b8b3cd") : secret "error-404-isvc-a3a91-predictor-serving-cert" not found
Jul 16 22:13:13.397392 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.397298    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:13.474537 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.474502    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:13.550040 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.550000    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"]
Jul 16 22:13:13.553956 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:13:13.553926    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6de4c344_fd98_4d3d_a0d8_04cdd26a394a.slice/crio-35a3b01bea8e1c159fe9b08f1d024c3e5c2c135bf27f6dfca724f58aa5e58db0 WatchSource:0}: Error finding container 35a3b01bea8e1c159fe9b08f1d024c3e5c2c135bf27f6dfca724f58aa5e58db0: Status 404 returned error can't find the container with id 35a3b01bea8e1c159fe9b08f1d024c3e5c2c135bf27f6dfca724f58aa5e58db0
Jul 16 22:13:13.963196 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:13.963071    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 16 22:13:14.163244 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.163206    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" event={"ID":"6de4c344-fd98-4d3d-a0d8-04cdd26a394a","Type":"ContainerStarted","Data":"6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232"}
Jul 16 22:13:14.163395 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.163299    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" event={"ID":"6de4c344-fd98-4d3d-a0d8-04cdd26a394a","Type":"ContainerStarted","Data":"aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17"}
Jul 16 22:13:14.163395 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.163329    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" event={"ID":"6de4c344-fd98-4d3d-a0d8-04cdd26a394a","Type":"ContainerStarted","Data":"35a3b01bea8e1c159fe9b08f1d024c3e5c2c135bf27f6dfca724f58aa5e58db0"}
Jul 16 22:13:14.163395 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.163352    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:14.182148 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.182065    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podStartSLOduration=2.182047949 podStartE2EDuration="2.182047949s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:13:14.180153701 +0000 UTC m=+1393.984577027" watchObservedRunningTime="2026-07-16 22:13:14.182047949 +0000 UTC m=+1393.986471275"
Jul 16 22:13:14.340570 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.340528    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:14.343161 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.343133    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls\") pod \"error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") " pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:14.435017 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.434979    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:14.587596 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.587570    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"]
Jul 16 22:13:14.589865 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:13:14.589831    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6591ac58_f4a1_4d2b_ab1e_653071b8b3cd.slice/crio-395204e8ab7c8dd0efabecd99ba77546e0c1d4f10e0d170dd31003bd35d04fd7 WatchSource:0}: Error finding container 395204e8ab7c8dd0efabecd99ba77546e0c1d4f10e0d170dd31003bd35d04fd7: Status 404 returned error can't find the container with id 395204e8ab7c8dd0efabecd99ba77546e0c1d4f10e0d170dd31003bd35d04fd7
Jul 16 22:13:14.967849 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:14.967735    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 16 22:13:15.170555 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:15.170517    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" event={"ID":"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd","Type":"ContainerStarted","Data":"d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8"}
Jul 16 22:13:15.170711 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:15.170588    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" event={"ID":"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd","Type":"ContainerStarted","Data":"cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e"}
Jul 16 22:13:15.170711 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:15.170612    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" event={"ID":"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd","Type":"ContainerStarted","Data":"395204e8ab7c8dd0efabecd99ba77546e0c1d4f10e0d170dd31003bd35d04fd7"}
Jul 16 22:13:15.170817 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:15.170767    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:15.172365 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:15.172335    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 16 22:13:15.191825 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:15.191767    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podStartSLOduration=3.191747117 podStartE2EDuration="3.191747117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:13:15.189485857 +0000 UTC m=+1394.993909213" watchObservedRunningTime="2026-07-16 22:13:15.191747117 +0000 UTC m=+1394.996170442"
Jul 16 22:13:15.702591 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:15.702544    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.48:8643/healthz\": dial tcp 10.134.0.48:8643: connect: connection refused"
Jul 16 22:13:15.703022 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:15.702544    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.47:8643/healthz\": dial tcp 10.134.0.47:8643: connect: connection refused"
Jul 16 22:13:16.175606 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.175556    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 16 22:13:16.175812 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.175664    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:16.175812 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.175734    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:16.177228 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.177196    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 16 22:13:16.618132 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.618069    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:13:16.664855 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.664816    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25fc5316-4d1e-480a-a6e4-db45339a8d09-proxy-tls\") pod \"25fc5316-4d1e-480a-a6e4-db45339a8d09\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") "
Jul 16 22:13:16.665030 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.664921    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9z4t\" (UniqueName: \"kubernetes.io/projected/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-api-access-b9z4t\") pod \"25fc5316-4d1e-480a-a6e4-db45339a8d09\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") "
Jul 16 22:13:16.665030 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.665003    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-rbac-proxy-sar-config\") pod \"25fc5316-4d1e-480a-a6e4-db45339a8d09\" (UID: \"25fc5316-4d1e-480a-a6e4-db45339a8d09\") "
Jul 16 22:13:16.665373 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.665350    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "25fc5316-4d1e-480a-a6e4-db45339a8d09" (UID: "25fc5316-4d1e-480a-a6e4-db45339a8d09"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:13:16.667180 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.667152    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-api-access-b9z4t" (OuterVolumeSpecName: "kube-api-access-b9z4t") pod "25fc5316-4d1e-480a-a6e4-db45339a8d09" (UID: "25fc5316-4d1e-480a-a6e4-db45339a8d09"). InnerVolumeSpecName "kube-api-access-b9z4t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:13:16.667294 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.667153    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25fc5316-4d1e-480a-a6e4-db45339a8d09-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "25fc5316-4d1e-480a-a6e4-db45339a8d09" (UID: "25fc5316-4d1e-480a-a6e4-db45339a8d09"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:13:16.766805 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.766755    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/25fc5316-4d1e-480a-a6e4-db45339a8d09-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:13:16.766805 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.766804    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9z4t\" (UniqueName: \"kubernetes.io/projected/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-api-access-b9z4t\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:13:16.766805 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:16.766816    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/25fc5316-4d1e-480a-a6e4-db45339a8d09-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:13:17.072756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.072729    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:13:17.170864 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.170817    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxrkm\" (UniqueName: \"kubernetes.io/projected/8f981066-32e7-46af-b68c-6a64857028b8-kube-api-access-gxrkm\") pod \"8f981066-32e7-46af-b68c-6a64857028b8\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") "
Jul 16 22:13:17.171072 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.170922    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls\") pod \"8f981066-32e7-46af-b68c-6a64857028b8\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") "
Jul 16 22:13:17.171072 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.171036    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f981066-32e7-46af-b68c-6a64857028b8-kube-rbac-proxy-sar-config\") pod \"8f981066-32e7-46af-b68c-6a64857028b8\" (UID: \"8f981066-32e7-46af-b68c-6a64857028b8\") "
Jul 16 22:13:17.171473 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.171447    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f981066-32e7-46af-b68c-6a64857028b8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "8f981066-32e7-46af-b68c-6a64857028b8" (UID: "8f981066-32e7-46af-b68c-6a64857028b8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:13:17.173111 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.173080    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "8f981066-32e7-46af-b68c-6a64857028b8" (UID: "8f981066-32e7-46af-b68c-6a64857028b8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:13:17.173191 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.173135    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f981066-32e7-46af-b68c-6a64857028b8-kube-api-access-gxrkm" (OuterVolumeSpecName: "kube-api-access-gxrkm") pod "8f981066-32e7-46af-b68c-6a64857028b8" (UID: "8f981066-32e7-46af-b68c-6a64857028b8"). InnerVolumeSpecName "kube-api-access-gxrkm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:13:17.183465 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.183433    2585 generic.go:356] "Generic (PLEG): container finished" podID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerID="df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4" exitCode=0
Jul 16 22:13:17.183595 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.183495    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"
Jul 16 22:13:17.183595 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.183519    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" event={"ID":"25fc5316-4d1e-480a-a6e4-db45339a8d09","Type":"ContainerDied","Data":"df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4"}
Jul 16 22:13:17.183595 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.183593    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs" event={"ID":"25fc5316-4d1e-480a-a6e4-db45339a8d09","Type":"ContainerDied","Data":"a4e2e4a83ba1b2152f27876f1d10d61675a6764f4df0cdc9e5699b1dedb98df4"}
Jul 16 22:13:17.183741 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.183626    2585 scope.go:117] "RemoveContainer" containerID="fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0"
Jul 16 22:13:17.185409 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.185386    2585 generic.go:356] "Generic (PLEG): container finished" podID="8f981066-32e7-46af-b68c-6a64857028b8" containerID="d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b" exitCode=0
Jul 16 22:13:17.185529 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.185453    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"
Jul 16 22:13:17.185529 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.185482    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" event={"ID":"8f981066-32e7-46af-b68c-6a64857028b8","Type":"ContainerDied","Data":"d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b"}
Jul 16 22:13:17.185643 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.185569    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x" event={"ID":"8f981066-32e7-46af-b68c-6a64857028b8","Type":"ContainerDied","Data":"9d230b27ba402aaf2c378e8f23a3fe50ccce49e6b6eb1d54f6517f26024b0b3b"}
Jul 16 22:13:17.186134 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.186088    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 16 22:13:17.193527 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.193500    2585 scope.go:117] "RemoveContainer" containerID="df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4"
Jul 16 22:13:17.209380 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.209347    2585 scope.go:117] "RemoveContainer" containerID="fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0"
Jul 16 22:13:17.209798 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:17.209769    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0\": container with ID starting with fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0 not found: ID does not exist" containerID="fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0"
Jul 16 22:13:17.209892 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.209809    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0"} err="failed to get container status \"fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0\": rpc error: code = NotFound desc = could not find container \"fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0\": container with ID starting with fa6494227ba2a983a01fb37fe964ef9e52845a9047eefe5a61528b9a296a55f0 not found: ID does not exist"
Jul 16 22:13:17.209892 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.209836    2585 scope.go:117] "RemoveContainer" containerID="df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4"
Jul 16 22:13:17.210202 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:17.210169    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4\": container with ID starting with df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4 not found: ID does not exist" containerID="df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4"
Jul 16 22:13:17.210315 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.210206    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4"} err="failed to get container status \"df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4\": rpc error: code = NotFound desc = could not find container \"df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4\": container with ID starting with df349d640b607243cf00e5dcca01ad85aacc0dc2818448c49461453e32c759f4 not found: ID does not exist"
Jul 16 22:13:17.210315 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.210229    2585 scope.go:117] "RemoveContainer" containerID="6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99"
Jul 16 22:13:17.220201 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.220066    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"]
Jul 16 22:13:17.222575 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.222537    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-67446-predictor-86c85bf98f-wwgzs"]
Jul 16 22:13:17.223318 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.223299    2585 scope.go:117] "RemoveContainer" containerID="d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b"
Jul 16 22:13:17.231979 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.231853    2585 scope.go:117] "RemoveContainer" containerID="6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99"
Jul 16 22:13:17.232519 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:17.232489    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99\": container with ID starting with 6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99 not found: ID does not exist" containerID="6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99"
Jul 16 22:13:17.232624 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.232530    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99"} err="failed to get container status \"6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99\": rpc error: code = NotFound desc = could not find container \"6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99\": container with ID starting with 6f2659bbb43a54c82ae7bfa05a7bdcca1b6bd35e5de64bedf530bfc6acef3c99 not found: ID does not exist"
Jul 16 22:13:17.232624 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.232550    2585 scope.go:117] "RemoveContainer" containerID="d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b"
Jul 16 22:13:17.232860 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:13:17.232840    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b\": container with ID starting with d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b not found: ID does not exist" containerID="d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b"
Jul 16 22:13:17.232918 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.232867    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b"} err="failed to get container status \"d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b\": rpc error: code = NotFound desc = could not find container \"d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b\": container with ID starting with d63877cb25a9b859dade6c32fe8f684762ca5b9e683cf2179096cce54bbfc42b not found: ID does not exist"
Jul 16 22:13:17.234851 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.234830    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"]
Jul 16 22:13:17.239205 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.239173    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-67446-predictor-58758849c6-zml4x"]
Jul 16 22:13:17.272650 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.272614    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/8f981066-32e7-46af-b68c-6a64857028b8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:13:17.272650 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.272650    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxrkm\" (UniqueName: \"kubernetes.io/projected/8f981066-32e7-46af-b68c-6a64857028b8-kube-api-access-gxrkm\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:13:17.272873 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:17.272667    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/8f981066-32e7-46af-b68c-6a64857028b8-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:13:18.827683 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:18.827640    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" path="/var/lib/kubelet/pods/25fc5316-4d1e-480a-a6e4-db45339a8d09/volumes"
Jul 16 22:13:18.828140 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:18.828125    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8f981066-32e7-46af-b68c-6a64857028b8" path="/var/lib/kubelet/pods/8f981066-32e7-46af-b68c-6a64857028b8/volumes"
Jul 16 22:13:21.180808 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:21.180777    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:13:21.181267 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:21.181242    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 16 22:13:22.192447 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:22.192416    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:13:22.193122 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:22.193068    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 16 22:13:23.963109 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:23.963048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 16 22:13:24.968344 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:24.968299    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 16 22:13:31.182232 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:31.182191    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 16 22:13:32.193784 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:32.193740    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 16 22:13:33.963266 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:33.963230    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:13:34.968308 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:34.968276    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:13:41.182223 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:41.182178    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 16 22:13:42.193153 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:42.193075    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 16 22:13:46.616901 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:46.616860    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:47.981349 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:47.981311    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:13:51.181305 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:51.181261    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 16 22:13:52.193733 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:13:52.193688    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 16 22:14:01.181736 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:14:01.181633    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 16 22:14:02.193951 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:14:02.193908    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.52:8080: connect: connection refused"
Jul 16 22:14:11.182296 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:14:11.182264    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:14:12.194972 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:14:12.194940    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:14:22.521174 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:14:22.521135    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:14:22.621581 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:14:22.621548    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:14:23.097461 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:14:23.097418    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:15:00.776999 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:15:00.776964    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:15:00.780477 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:15:00.780450    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:19:20.771305 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:19:20.771257    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:20:00.814850 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:20:00.814818    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:20:00.818459 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:20:00.818434    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:22:01.294570 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.294519    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:01.384727 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.384680    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"]
Jul 16 22:22:01.385046 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.385004    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" containerID="cri-o://bec91a44e26a78cccf0185467b9bc44589c87146c1992c98fe8ceda7ec271242" gracePeriod=30
Jul 16 22:22:01.385229 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.385056    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kube-rbac-proxy" containerID="cri-o://a1d0601393e8e135ef9f9768bd059b0b02f91ad1512c856b74996117c8c999c3" gracePeriod=30
Jul 16 22:22:01.416412 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.416379    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:01.502744 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.502714    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"]
Jul 16 22:22:01.503470 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503418    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kube-rbac-proxy"
Jul 16 22:22:01.503470 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503441    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kube-rbac-proxy"
Jul 16 22:22:01.503470 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503460    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container"
Jul 16 22:22:01.503470 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503470    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container"
Jul 16 22:22:01.503774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503492    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kube-rbac-proxy"
Jul 16 22:22:01.503774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503501    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kube-rbac-proxy"
Jul 16 22:22:01.503774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503517    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container"
Jul 16 22:22:01.503774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503525    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container"
Jul 16 22:22:01.503774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503625    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kserve-container"
Jul 16 22:22:01.503774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503645    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kserve-container"
Jul 16 22:22:01.503774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503658    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="25fc5316-4d1e-480a-a6e4-db45339a8d09" containerName="kube-rbac-proxy"
Jul 16 22:22:01.503774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.503669    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="8f981066-32e7-46af-b68c-6a64857028b8" containerName="kube-rbac-proxy"
Jul 16 22:22:01.507766 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.507740    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.513360 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.513329    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a3d07-predictor-serving-cert\""
Jul 16 22:22:01.513540 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.513329    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-a3d07-kube-rbac-proxy-sar-config\""
Jul 16 22:22:01.524577 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.524490    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"]
Jul 16 22:22:01.547880 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.547783    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:01.563658 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.563624    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"]
Jul 16 22:22:01.563921 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.563874    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" containerID="cri-o://11214c5bebf4a27d07356dde35e626614d40e1c8cc77b0bb5920b1d04f98ea04" gracePeriod=30
Jul 16 22:22:01.564006 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.563894    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kube-rbac-proxy" containerID="cri-o://d91b7016082b787872f46e81f857660dc971f06baf8c0f7955d234a72e04628b" gracePeriod=30
Jul 16 22:22:01.618262 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.618222    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"]
Jul 16 22:22:01.622182 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.622154    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.629981 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.629952    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3d07-predictor-serving-cert\""
Jul 16 22:22:01.630129 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.629981    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-a3d07-kube-rbac-proxy-sar-config\""
Jul 16 22:22:01.638080 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.638051    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"]
Jul 16 22:22:01.659653 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.659595    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c72fb\" (UniqueName: \"kubernetes.io/projected/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-api-access-c72fb\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.659832 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.659794    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.659891 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.659861    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a885f6f-87ef-4ca3-8830-d272bb0a0127-proxy-tls\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.678254 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.678224    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:01.761043 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.761005    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.761273 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.761056    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jxsrp\" (UniqueName: \"kubernetes.io/projected/653062fd-7a45-4f57-a939-69400a3be794-kube-api-access-jxsrp\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.761273 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.761084    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a885f6f-87ef-4ca3-8830-d272bb0a0127-proxy-tls\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.761273 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.761148    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653062fd-7a45-4f57-a939-69400a3be794-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.761452 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.761301    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c72fb\" (UniqueName: \"kubernetes.io/projected/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-api-access-c72fb\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.761452 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.761382    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653062fd-7a45-4f57-a939-69400a3be794-proxy-tls\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.761794 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.761773    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.763632 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.763608    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a885f6f-87ef-4ca3-8830-d272bb0a0127-proxy-tls\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.770695 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.770670    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c72fb\" (UniqueName: \"kubernetes.io/projected/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-api-access-c72fb\") pod \"success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") " pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.789739 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.789704    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:01.822538 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.822441    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:01.862826 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.862778    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653062fd-7a45-4f57-a939-69400a3be794-proxy-tls\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.863140 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.862876    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jxsrp\" (UniqueName: \"kubernetes.io/projected/653062fd-7a45-4f57-a939-69400a3be794-kube-api-access-jxsrp\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.863140 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.862936    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653062fd-7a45-4f57-a939-69400a3be794-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.864026 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.863993    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653062fd-7a45-4f57-a939-69400a3be794-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.865837 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.865813    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653062fd-7a45-4f57-a939-69400a3be794-proxy-tls\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.872675 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.872648    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jxsrp\" (UniqueName: \"kubernetes.io/projected/653062fd-7a45-4f57-a939-69400a3be794-kube-api-access-jxsrp\") pod \"error-404-isvc-a3d07-predictor-f5d955f89-cthdq\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") " pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.933185 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.933146    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:01.951065 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.951024    2585 generic.go:356] "Generic (PLEG): container finished" podID="86a47779-330b-4d20-87ba-123ae324eea8" containerID="d91b7016082b787872f46e81f857660dc971f06baf8c0f7955d234a72e04628b" exitCode=2
Jul 16 22:22:01.951260 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.951236    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" event={"ID":"86a47779-330b-4d20-87ba-123ae324eea8","Type":"ContainerDied","Data":"d91b7016082b787872f46e81f857660dc971f06baf8c0f7955d234a72e04628b"}
Jul 16 22:22:01.953631 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.953598    2585 generic.go:356] "Generic (PLEG): container finished" podID="56149faf-9310-48bd-b667-2f9e9216898e" containerID="a1d0601393e8e135ef9f9768bd059b0b02f91ad1512c856b74996117c8c999c3" exitCode=2
Jul 16 22:22:01.953787 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.953658    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" event={"ID":"56149faf-9310-48bd-b667-2f9e9216898e","Type":"ContainerDied","Data":"a1d0601393e8e135ef9f9768bd059b0b02f91ad1512c856b74996117c8c999c3"}
Jul 16 22:22:01.977657 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:01.977628    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"]
Jul 16 22:22:01.979797 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:22:01.979770    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a885f6f_87ef_4ca3_8830_d272bb0a0127.slice/crio-3e40afc2cb3d8313eac52ff264dd4513db410e9351ba9f05bfde26a5925b2dab WatchSource:0}: Error finding container 3e40afc2cb3d8313eac52ff264dd4513db410e9351ba9f05bfde26a5925b2dab: Status 404 returned error can't find the container with id 3e40afc2cb3d8313eac52ff264dd4513db410e9351ba9f05bfde26a5925b2dab
Jul 16 22:22:02.101474 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.101443    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"]
Jul 16 22:22:02.103349 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:22:02.103310    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653062fd_7a45_4f57_a939_69400a3be794.slice/crio-a4a02cb4feaf67d2404d7bac6da6e45a5a07a467004429c722e1cfe4bd48db50 WatchSource:0}: Error finding container a4a02cb4feaf67d2404d7bac6da6e45a5a07a467004429c722e1cfe4bd48db50: Status 404 returned error can't find the container with id a4a02cb4feaf67d2404d7bac6da6e45a5a07a467004429c722e1cfe4bd48db50
Jul 16 22:22:02.961757 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.961722    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" event={"ID":"653062fd-7a45-4f57-a939-69400a3be794","Type":"ContainerStarted","Data":"1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a"}
Jul 16 22:22:02.962269 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.961793    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" event={"ID":"653062fd-7a45-4f57-a939-69400a3be794","Type":"ContainerStarted","Data":"f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5"}
Jul 16 22:22:02.962269 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.961823    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" event={"ID":"653062fd-7a45-4f57-a939-69400a3be794","Type":"ContainerStarted","Data":"a4a02cb4feaf67d2404d7bac6da6e45a5a07a467004429c722e1cfe4bd48db50"}
Jul 16 22:22:02.962269 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.961854    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:02.962269 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.961886    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:02.963620 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.963578    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 16 22:22:02.963802 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.963781    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" event={"ID":"6a885f6f-87ef-4ca3-8830-d272bb0a0127","Type":"ContainerStarted","Data":"df30702306dec8328da9d82e2b4a2dc9b4675041daa91d86cab90372e576ad9e"}
Jul 16 22:22:02.963895 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.963827    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" event={"ID":"6a885f6f-87ef-4ca3-8830-d272bb0a0127","Type":"ContainerStarted","Data":"d64ce63f1ffcc25a8b0d2edf11828f8ebbce4a0da47b68bdaf68574664d536dd"}
Jul 16 22:22:02.963895 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.963851    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:02.963895 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.963872    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" event={"ID":"6a885f6f-87ef-4ca3-8830-d272bb0a0127","Type":"ContainerStarted","Data":"3e40afc2cb3d8313eac52ff264dd4513db410e9351ba9f05bfde26a5925b2dab"}
Jul 16 22:22:02.983332 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:02.983265    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podStartSLOduration=1.9832471219999999 podStartE2EDuration="1.983247122s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:02.981093578 +0000 UTC m=+1922.785516905" watchObservedRunningTime="2026-07-16 22:22:02.983247122 +0000 UTC m=+1922.787670449"
Jul 16 22:22:03.003037 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:03.002977    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podStartSLOduration=2.002956192 podStartE2EDuration="2.002956192s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:03.00091672 +0000 UTC m=+1922.805340060" watchObservedRunningTime="2026-07-16 22:22:03.002956192 +0000 UTC m=+1922.807379517"
Jul 16 22:22:03.956826 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:03.956776    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.50:8643/healthz\": dial tcp 10.134.0.50:8643: connect: connection refused"
Jul 16 22:22:03.962303 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:03.962257    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.50:8080: connect: connection refused"
Jul 16 22:22:03.969029 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:03.968998    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:03.969248 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:03.969192    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 16 22:22:03.970671 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:03.970644    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 16 22:22:04.962958 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:04.962907    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.49:8643/healthz\": dial tcp 10.134.0.49:8643: connect: connection refused"
Jul 16 22:22:04.968408 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:04.968376    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.49:8080: connect: connection refused"
Jul 16 22:22:04.974322 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:04.974283    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 16 22:22:05.983762 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:05.983324    2585 generic.go:356] "Generic (PLEG): container finished" podID="86a47779-330b-4d20-87ba-123ae324eea8" containerID="11214c5bebf4a27d07356dde35e626614d40e1c8cc77b0bb5920b1d04f98ea04" exitCode=0
Jul 16 22:22:05.983762 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:05.983382    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" event={"ID":"86a47779-330b-4d20-87ba-123ae324eea8","Type":"ContainerDied","Data":"11214c5bebf4a27d07356dde35e626614d40e1c8cc77b0bb5920b1d04f98ea04"}
Jul 16 22:22:05.986681 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:05.986646    2585 generic.go:356] "Generic (PLEG): container finished" podID="56149faf-9310-48bd-b667-2f9e9216898e" containerID="bec91a44e26a78cccf0185467b9bc44589c87146c1992c98fe8ceda7ec271242" exitCode=0
Jul 16 22:22:05.986824 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:05.986683    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" event={"ID":"56149faf-9310-48bd-b667-2f9e9216898e","Type":"ContainerDied","Data":"bec91a44e26a78cccf0185467b9bc44589c87146c1992c98fe8ceda7ec271242"}
Jul 16 22:22:06.062108 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.062061    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:22:06.115536 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.115506    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7q774\" (UniqueName: \"kubernetes.io/projected/56149faf-9310-48bd-b667-2f9e9216898e-kube-api-access-7q774\") pod \"56149faf-9310-48bd-b667-2f9e9216898e\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") "
Jul 16 22:22:06.115706 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.115602    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56149faf-9310-48bd-b667-2f9e9216898e-kube-rbac-proxy-sar-config\") pod \"56149faf-9310-48bd-b667-2f9e9216898e\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") "
Jul 16 22:22:06.115706 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.115667    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56149faf-9310-48bd-b667-2f9e9216898e-proxy-tls\") pod \"56149faf-9310-48bd-b667-2f9e9216898e\" (UID: \"56149faf-9310-48bd-b667-2f9e9216898e\") "
Jul 16 22:22:06.116046 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.116009    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56149faf-9310-48bd-b667-2f9e9216898e-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "56149faf-9310-48bd-b667-2f9e9216898e" (UID: "56149faf-9310-48bd-b667-2f9e9216898e"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:06.117944 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.117909    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56149faf-9310-48bd-b667-2f9e9216898e-kube-api-access-7q774" (OuterVolumeSpecName: "kube-api-access-7q774") pod "56149faf-9310-48bd-b667-2f9e9216898e" (UID: "56149faf-9310-48bd-b667-2f9e9216898e"). InnerVolumeSpecName "kube-api-access-7q774". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:22:06.118074 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.117984    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/56149faf-9310-48bd-b667-2f9e9216898e-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "56149faf-9310-48bd-b667-2f9e9216898e" (UID: "56149faf-9310-48bd-b667-2f9e9216898e"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:22:06.123251 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.123229    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:22:06.216454 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.216420    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a47779-330b-4d20-87ba-123ae324eea8-proxy-tls\") pod \"86a47779-330b-4d20-87ba-123ae324eea8\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") "
Jul 16 22:22:06.216639 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.216497    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a47779-330b-4d20-87ba-123ae324eea8-kube-rbac-proxy-sar-config\") pod \"86a47779-330b-4d20-87ba-123ae324eea8\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") "
Jul 16 22:22:06.216639 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.216520    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5525h\" (UniqueName: \"kubernetes.io/projected/86a47779-330b-4d20-87ba-123ae324eea8-kube-api-access-5525h\") pod \"86a47779-330b-4d20-87ba-123ae324eea8\" (UID: \"86a47779-330b-4d20-87ba-123ae324eea8\") "
Jul 16 22:22:06.216819 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.216791    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/56149faf-9310-48bd-b667-2f9e9216898e-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:06.216899 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.216818    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7q774\" (UniqueName: \"kubernetes.io/projected/56149faf-9310-48bd-b667-2f9e9216898e-kube-api-access-7q774\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:06.216899 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.216835    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/56149faf-9310-48bd-b667-2f9e9216898e-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:06.216973 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.216912    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86a47779-330b-4d20-87ba-123ae324eea8-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "86a47779-330b-4d20-87ba-123ae324eea8" (UID: "86a47779-330b-4d20-87ba-123ae324eea8"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:06.218789 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.218759    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86a47779-330b-4d20-87ba-123ae324eea8-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "86a47779-330b-4d20-87ba-123ae324eea8" (UID: "86a47779-330b-4d20-87ba-123ae324eea8"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:22:06.218910 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.218767    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86a47779-330b-4d20-87ba-123ae324eea8-kube-api-access-5525h" (OuterVolumeSpecName: "kube-api-access-5525h") pod "86a47779-330b-4d20-87ba-123ae324eea8" (UID: "86a47779-330b-4d20-87ba-123ae324eea8"). InnerVolumeSpecName "kube-api-access-5525h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:22:06.317669 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.317621    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/86a47779-330b-4d20-87ba-123ae324eea8-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:06.317669 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.317669    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5525h\" (UniqueName: \"kubernetes.io/projected/86a47779-330b-4d20-87ba-123ae324eea8-kube-api-access-5525h\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:06.317887 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.317686    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/86a47779-330b-4d20-87ba-123ae324eea8-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:06.993592 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.993563    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"
Jul 16 22:22:06.994122 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.993598    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr" event={"ID":"56149faf-9310-48bd-b667-2f9e9216898e","Type":"ContainerDied","Data":"f8f06c3abdffa6f6d9b27b5f946a11bf8f4afc6e40385b41d2f93d1765f4927e"}
Jul 16 22:22:06.994122 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.993677    2585 scope.go:117] "RemoveContainer" containerID="a1d0601393e8e135ef9f9768bd059b0b02f91ad1512c856b74996117c8c999c3"
Jul 16 22:22:06.995584 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.995562    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"
Jul 16 22:22:06.995584 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:06.995569    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5" event={"ID":"86a47779-330b-4d20-87ba-123ae324eea8","Type":"ContainerDied","Data":"d0f1e47e9deca5352967411d27cff8ab47306ba5204c2bcd7adb0dbf5a683ec4"}
Jul 16 22:22:07.004129 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:07.004089    2585 scope.go:117] "RemoveContainer" containerID="bec91a44e26a78cccf0185467b9bc44589c87146c1992c98fe8ceda7ec271242"
Jul 16 22:22:07.016657 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:07.016622    2585 scope.go:117] "RemoveContainer" containerID="d91b7016082b787872f46e81f857660dc971f06baf8c0f7955d234a72e04628b"
Jul 16 22:22:07.020724 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:07.020690    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"]
Jul 16 22:22:07.027903 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:07.027879    2585 scope.go:117] "RemoveContainer" containerID="11214c5bebf4a27d07356dde35e626614d40e1c8cc77b0bb5920b1d04f98ea04"
Jul 16 22:22:07.030153 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:07.030115    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-c1850-predictor-597c44d6bf-wwcv5"]
Jul 16 22:22:07.047166 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:07.047135    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"]
Jul 16 22:22:07.052914 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:07.052882    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-c1850-predictor-7f7fd6c8cb-z78wr"]
Jul 16 22:22:08.827067 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:08.827025    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="56149faf-9310-48bd-b667-2f9e9216898e" path="/var/lib/kubelet/pods/56149faf-9310-48bd-b667-2f9e9216898e/volumes"
Jul 16 22:22:08.834447 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:08.827948    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="86a47779-330b-4d20-87ba-123ae324eea8" path="/var/lib/kubelet/pods/86a47779-330b-4d20-87ba-123ae324eea8/volumes"
Jul 16 22:22:08.973497 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:08.973466    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:08.974019 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:08.973981    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 16 22:22:09.979746 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:09.979708    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:22:09.980362 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:09.980330    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 16 22:22:18.974496 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:18.974449    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 16 22:22:19.980552 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:19.980509    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 16 22:22:28.974385 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:28.974336    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 16 22:22:29.980607 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:29.980566    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 16 22:22:37.243366 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.243327    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:37.375655 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.375624    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"]
Jul 16 22:22:37.375916 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.375888    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" containerID="cri-o://aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17" gracePeriod=30
Jul 16 22:22:37.375982 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.375935    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kube-rbac-proxy" containerID="cri-o://6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232" gracePeriod=30
Jul 16 22:22:37.441612 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.441573    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:37.454823 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.454780    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"]
Jul 16 22:22:37.455332 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455312    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container"
Jul 16 22:22:37.455423 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455335    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container"
Jul 16 22:22:37.455423 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455356    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kube-rbac-proxy"
Jul 16 22:22:37.455423 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455365    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kube-rbac-proxy"
Jul 16 22:22:37.455423 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455403    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kube-rbac-proxy"
Jul 16 22:22:37.455423 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455411    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kube-rbac-proxy"
Jul 16 22:22:37.455423 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455425    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container"
Jul 16 22:22:37.455622 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455431    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container"
Jul 16 22:22:37.455622 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455540    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kube-rbac-proxy"
Jul 16 22:22:37.455622 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455559    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="56149faf-9310-48bd-b667-2f9e9216898e" containerName="kserve-container"
Jul 16 22:22:37.455622 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455570    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kube-rbac-proxy"
Jul 16 22:22:37.455622 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.455585    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="86a47779-330b-4d20-87ba-123ae324eea8" containerName="kserve-container"
Jul 16 22:22:37.458970 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.458944    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.462352 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.462322    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-93462-predictor-serving-cert\""
Jul 16 22:22:37.462494 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.462387    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-93462-kube-rbac-proxy-sar-config\""
Jul 16 22:22:37.484122 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.484060    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"]
Jul 16 22:22:37.563752 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.563706    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"]
Jul 16 22:22:37.564004 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.563979    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container" containerID="cri-o://cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e" gracePeriod=30
Jul 16 22:22:37.564149 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.564029    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kube-rbac-proxy" containerID="cri-o://d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8" gracePeriod=30
Jul 16 22:22:37.579588 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:22:37.579546    2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6591ac58_f4a1_4d2b_ab1e_653071b8b3cd.slice/crio-conmon-d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6591ac58_f4a1_4d2b_ab1e_653071b8b3cd.slice/crio-d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8.scope\": RecentStats: unable to find data in memory cache]"
Jul 16 22:22:37.608877 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.608827    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:37.620185 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.620139    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.620361 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.620265    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.620361 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.620347    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6zp7\" (UniqueName: \"kubernetes.io/projected/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-api-access-k6zp7\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.634467 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.634432    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"]
Jul 16 22:22:37.638408 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.638389    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.640866 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.640838    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-93462-predictor-serving-cert\""
Jul 16 22:22:37.641253 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.641233    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-93462-kube-rbac-proxy-sar-config\""
Jul 16 22:22:37.676920 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.676883    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"]
Jul 16 22:22:37.721653 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.721619    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.721946 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.721685    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.721946 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.721757    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6zp7\" (UniqueName: \"kubernetes.io/projected/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-api-access-k6zp7\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.721946 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:22:37.721851    2585 secret.go:189] Couldn't get secret kserve-ci-e2e-test/success-200-isvc-93462-predictor-serving-cert: secret "success-200-isvc-93462-predictor-serving-cert" not found
Jul 16 22:22:37.721946 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:22:37.721930    2585 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls podName:3e2c3c09-dd80-4362-9c24-6b7bb097e018 nodeName:}" failed. No retries permitted until 2026-07-16 22:22:38.221908644 +0000 UTC m=+1958.026331961 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "proxy-tls" (UniqueName: "kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls") pod "success-200-isvc-93462-predictor-759cb758b6-4cnb7" (UID: "3e2c3c09-dd80-4362-9c24-6b7bb097e018") : secret "success-200-isvc-93462-predictor-serving-cert" not found
Jul 16 22:22:37.722439 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.722417    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.724131 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.724091    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:37.733839 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.733808    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6zp7\" (UniqueName: \"kubernetes.io/projected/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-api-access-k6zp7\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:37.822322 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.822230    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b91099-75f0-47a7-994a-fba7ebdfd400-proxy-tls\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.822322 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.822284    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6dgs4\" (UniqueName: \"kubernetes.io/projected/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-api-access-6dgs4\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.822322 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.822306    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.832818 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.832779    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:37.924008 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.923964    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6dgs4\" (UniqueName: \"kubernetes.io/projected/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-api-access-6dgs4\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.924008 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.924008    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.924254 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.924160    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b91099-75f0-47a7-994a-fba7ebdfd400-proxy-tls\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.924715 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.924686    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.926593 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.926563    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b91099-75f0-47a7-994a-fba7ebdfd400-proxy-tls\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.937797 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.937764    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6dgs4\" (UniqueName: \"kubernetes.io/projected/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-api-access-6dgs4\") pod \"error-404-isvc-93462-predictor-655689f48c-4gb5h\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") " pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.949730 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.949684    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:37.972951 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:37.972921    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:22:38.103050 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.103015    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"]
Jul 16 22:22:38.104486 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:22:38.104459    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97b91099_75f0_47a7_994a_fba7ebdfd400.slice/crio-f603ed17924dc3248bbd3dd5abed336442777134087ed6d606ac13f0cb31c4ba WatchSource:0}: Error finding container f603ed17924dc3248bbd3dd5abed336442777134087ed6d606ac13f0cb31c4ba: Status 404 returned error can't find the container with id f603ed17924dc3248bbd3dd5abed336442777134087ed6d606ac13f0cb31c4ba
Jul 16 22:22:38.172003 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.171977    2585 generic.go:356] "Generic (PLEG): container finished" podID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerID="6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232" exitCode=2
Jul 16 22:22:38.172144 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.172048    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" event={"ID":"6de4c344-fd98-4d3d-a0d8-04cdd26a394a","Type":"ContainerDied","Data":"6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232"}
Jul 16 22:22:38.173906 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.173886    2585 generic.go:356] "Generic (PLEG): container finished" podID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerID="d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8" exitCode=2
Jul 16 22:22:38.173978 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.173933    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" event={"ID":"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd","Type":"ContainerDied","Data":"d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8"}
Jul 16 22:22:38.175629 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.175601    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" event={"ID":"97b91099-75f0-47a7-994a-fba7ebdfd400","Type":"ContainerStarted","Data":"4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290"}
Jul 16 22:22:38.175727 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.175644    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" event={"ID":"97b91099-75f0-47a7-994a-fba7ebdfd400","Type":"ContainerStarted","Data":"f603ed17924dc3248bbd3dd5abed336442777134087ed6d606ac13f0cb31c4ba"}
Jul 16 22:22:38.226455 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.226414    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:38.228782 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.228760    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls\") pod \"success-200-isvc-93462-predictor-759cb758b6-4cnb7\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") " pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:38.371005 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.370911    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:38.522092 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.522061    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"]
Jul 16 22:22:38.523476 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:22:38.523444    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e2c3c09_dd80_4362_9c24_6b7bb097e018.slice/crio-7b2fec4f683fa20d11ce3d7c72f8ab2da055db6fd0fcfb8c79453e5a30843264 WatchSource:0}: Error finding container 7b2fec4f683fa20d11ce3d7c72f8ab2da055db6fd0fcfb8c79453e5a30843264: Status 404 returned error can't find the container with id 7b2fec4f683fa20d11ce3d7c72f8ab2da055db6fd0fcfb8c79453e5a30843264
Jul 16 22:22:38.974536 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:38.974434    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 16 22:22:39.183626 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.183584    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" event={"ID":"97b91099-75f0-47a7-994a-fba7ebdfd400","Type":"ContainerStarted","Data":"b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3"}
Jul 16 22:22:39.183948 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.183737    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:39.185520 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.185492    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" event={"ID":"3e2c3c09-dd80-4362-9c24-6b7bb097e018","Type":"ContainerStarted","Data":"cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe"}
Jul 16 22:22:39.185648 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.185543    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" event={"ID":"3e2c3c09-dd80-4362-9c24-6b7bb097e018","Type":"ContainerStarted","Data":"cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7"}
Jul 16 22:22:39.185648 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.185569    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:39.185648 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.185588    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" event={"ID":"3e2c3c09-dd80-4362-9c24-6b7bb097e018","Type":"ContainerStarted","Data":"7b2fec4f683fa20d11ce3d7c72f8ab2da055db6fd0fcfb8c79453e5a30843264"}
Jul 16 22:22:39.185648 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.185609    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:39.187537 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.187506    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 16 22:22:39.207960 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.207903    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podStartSLOduration=2.207884424 podStartE2EDuration="2.207884424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:39.20605079 +0000 UTC m=+1959.010474129" watchObservedRunningTime="2026-07-16 22:22:39.207884424 +0000 UTC m=+1959.012307750"
Jul 16 22:22:39.233552 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.233468    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podStartSLOduration=2.233447531 podStartE2EDuration="2.233447531s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:22:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:22:39.23177102 +0000 UTC m=+1959.036194393" watchObservedRunningTime="2026-07-16 22:22:39.233447531 +0000 UTC m=+1959.037870856"
Jul 16 22:22:39.980633 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:39.980592    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 16 22:22:40.191225 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:40.191176    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 16 22:22:40.191436 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:40.191286    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:40.192850 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:40.192817    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 16 22:22:41.176178 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:41.176133    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.51:8643/healthz\": dial tcp 10.134.0.51:8643: connect: connection refused"
Jul 16 22:22:41.181478 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:41.181440    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.51:8080: connect: connection refused"
Jul 16 22:22:41.196041 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:41.196000    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 16 22:22:41.974793 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:41.974724    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:22:41.978055 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:41.978029    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:22:42.164744 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.164706    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qm2zd\" (UniqueName: \"kubernetes.io/projected/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-api-access-qm2zd\") pod \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") "
Jul 16 22:22:42.164744 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.164752    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-rbac-proxy-sar-config\") pod \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") "
Jul 16 22:22:42.165007 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.164804    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fz2kt\" (UniqueName: \"kubernetes.io/projected/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-api-access-fz2kt\") pod \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") "
Jul 16 22:22:42.165007 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.164856    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls\") pod \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") "
Jul 16 22:22:42.165007 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.164895    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-rbac-proxy-sar-config\") pod \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\" (UID: \"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd\") "
Jul 16 22:22:42.165007 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.164921    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls\") pod \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\" (UID: \"6de4c344-fd98-4d3d-a0d8-04cdd26a394a\") "
Jul 16 22:22:42.165271 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.165184    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6de4c344-fd98-4d3d-a0d8-04cdd26a394a" (UID: "6de4c344-fd98-4d3d-a0d8-04cdd26a394a"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:42.165558 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.165536    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" (UID: "6591ac58-f4a1-4d2b-ab1e-653071b8b3cd"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:22:42.167231 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.167206    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-api-access-qm2zd" (OuterVolumeSpecName: "kube-api-access-qm2zd") pod "6de4c344-fd98-4d3d-a0d8-04cdd26a394a" (UID: "6de4c344-fd98-4d3d-a0d8-04cdd26a394a"). InnerVolumeSpecName "kube-api-access-qm2zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:22:42.167308 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.167240    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-api-access-fz2kt" (OuterVolumeSpecName: "kube-api-access-fz2kt") pod "6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" (UID: "6591ac58-f4a1-4d2b-ab1e-653071b8b3cd"). InnerVolumeSpecName "kube-api-access-fz2kt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:22:42.167449 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.167420    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" (UID: "6591ac58-f4a1-4d2b-ab1e-653071b8b3cd"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:22:42.167513 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.167464    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6de4c344-fd98-4d3d-a0d8-04cdd26a394a" (UID: "6de4c344-fd98-4d3d-a0d8-04cdd26a394a"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:22:42.202966 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.202924    2585 generic.go:356] "Generic (PLEG): container finished" podID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerID="aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17" exitCode=0
Jul 16 22:22:42.203421 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.202980    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"
Jul 16 22:22:42.203421 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.203002    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" event={"ID":"6de4c344-fd98-4d3d-a0d8-04cdd26a394a","Type":"ContainerDied","Data":"aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17"}
Jul 16 22:22:42.203421 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.203141    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7" event={"ID":"6de4c344-fd98-4d3d-a0d8-04cdd26a394a","Type":"ContainerDied","Data":"35a3b01bea8e1c159fe9b08f1d024c3e5c2c135bf27f6dfca724f58aa5e58db0"}
Jul 16 22:22:42.203421 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.203184    2585 scope.go:117] "RemoveContainer" containerID="6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232"
Jul 16 22:22:42.204975 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.204946    2585 generic.go:356] "Generic (PLEG): container finished" podID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerID="cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e" exitCode=0
Jul 16 22:22:42.205109 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.205009    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"
Jul 16 22:22:42.205109 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.205026    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" event={"ID":"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd","Type":"ContainerDied","Data":"cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e"}
Jul 16 22:22:42.205185 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.205092    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66" event={"ID":"6591ac58-f4a1-4d2b-ab1e-653071b8b3cd","Type":"ContainerDied","Data":"395204e8ab7c8dd0efabecd99ba77546e0c1d4f10e0d170dd31003bd35d04fd7"}
Jul 16 22:22:42.215393 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.214591    2585 scope.go:117] "RemoveContainer" containerID="aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17"
Jul 16 22:22:42.225167 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.225148    2585 scope.go:117] "RemoveContainer" containerID="6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232"
Jul 16 22:22:42.225464 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:22:42.225442    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232\": container with ID starting with 6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232 not found: ID does not exist" containerID="6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232"
Jul 16 22:22:42.225534 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.225472    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232"} err="failed to get container status \"6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232\": rpc error: code = NotFound desc = could not find container \"6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232\": container with ID starting with 6b23d31e451184a1ab12eefc49693168226306375a5e385e1164e25650da0232 not found: ID does not exist"
Jul 16 22:22:42.225534 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.225492    2585 scope.go:117] "RemoveContainer" containerID="aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17"
Jul 16 22:22:42.225720 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:22:42.225705    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17\": container with ID starting with aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17 not found: ID does not exist" containerID="aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17"
Jul 16 22:22:42.225756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.225724    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17"} err="failed to get container status \"aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17\": rpc error: code = NotFound desc = could not find container \"aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17\": container with ID starting with aa8a5e40822858e9419aacfe9907d5260e3bd0f4ed8320ebf120638e254b5d17 not found: ID does not exist"
Jul 16 22:22:42.225756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.225737    2585 scope.go:117] "RemoveContainer" containerID="d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8"
Jul 16 22:22:42.234831 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.234804    2585 scope.go:117] "RemoveContainer" containerID="cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e"
Jul 16 22:22:42.235216 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.235163    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"]
Jul 16 22:22:42.237954 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.237932    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3a91-predictor-d4f677b68-tggk7"]
Jul 16 22:22:42.245430 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.245405    2585 scope.go:117] "RemoveContainer" containerID="d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8"
Jul 16 22:22:42.245764 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:22:42.245743    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8\": container with ID starting with d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8 not found: ID does not exist" containerID="d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8"
Jul 16 22:22:42.245809 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.245774    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8"} err="failed to get container status \"d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8\": rpc error: code = NotFound desc = could not find container \"d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8\": container with ID starting with d6aa09654ca96e17408784b29b5f1930857ecdc05aa34c07e632cf1701ee37f8 not found: ID does not exist"
Jul 16 22:22:42.245809 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.245794    2585 scope.go:117] "RemoveContainer" containerID="cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e"
Jul 16 22:22:42.246082 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:22:42.246057    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e\": container with ID starting with cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e not found: ID does not exist" containerID="cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e"
Jul 16 22:22:42.246180 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.246091    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e"} err="failed to get container status \"cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e\": rpc error: code = NotFound desc = could not find container \"cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e\": container with ID starting with cebcc87ff164bd7256086ef3acf76a407a7fd2a37a7fd240cec4cdb7da530a2e not found: ID does not exist"
Jul 16 22:22:42.249138 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.249087    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"]
Jul 16 22:22:42.255385 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.255352    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3a91-predictor-844ff5cbfc-fnf66"]
Jul 16 22:22:42.266298 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.266261    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qm2zd\" (UniqueName: \"kubernetes.io/projected/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-api-access-qm2zd\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:42.266298 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.266292    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:42.266298 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.266305    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fz2kt\" (UniqueName: \"kubernetes.io/projected/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-api-access-fz2kt\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:42.266510 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.266314    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:42.266510 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.266324    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:42.266510 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.266332    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6de4c344-fd98-4d3d-a0d8-04cdd26a394a-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:22:42.828033 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.827984    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" path="/var/lib/kubelet/pods/6591ac58-f4a1-4d2b-ab1e-653071b8b3cd/volumes"
Jul 16 22:22:42.828581 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:42.828556    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" path="/var/lib/kubelet/pods/6de4c344-fd98-4d3d-a0d8-04cdd26a394a/volumes"
Jul 16 22:22:45.195613 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:45.195581    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:22:45.196368 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:45.196334    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 16 22:22:46.201521 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:46.201491    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:22:46.202002 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:46.201977    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 16 22:22:48.975008 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:48.974961    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.54:8080: connect: connection refused"
Jul 16 22:22:49.980568 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:49.980526    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.53:8080: connect: connection refused"
Jul 16 22:22:55.197208 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:55.197083    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 16 22:22:56.202930 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:56.202889    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 16 22:22:58.975963 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:58.975928    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:22:59.981815 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:22:59.981778    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:23:05.197073 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:05.197015    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 16 22:23:06.202124 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:06.202048    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 16 22:23:11.524516 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:11.524474    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:12.249008 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:12.248959    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:15.196539 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:15.196487    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 16 22:23:16.202181 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:16.202137    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 16 22:23:21.726581 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.726541    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:21.846623 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.846587    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"]
Jul 16 22:23:21.846917 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.846877    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kube-rbac-proxy" containerID="cri-o://df30702306dec8328da9d82e2b4a2dc9b4675041daa91d86cab90372e576ad9e" gracePeriod=30
Jul 16 22:23:21.847021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.846858    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container" containerID="cri-o://d64ce63f1ffcc25a8b0d2edf11828f8ebbce4a0da47b68bdaf68574664d536dd" gracePeriod=30
Jul 16 22:23:21.879758 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.879723    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:21.908190 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908138    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"]
Jul 16 22:23:21.908841 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908814    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kube-rbac-proxy"
Jul 16 22:23:21.908942 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908846    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kube-rbac-proxy"
Jul 16 22:23:21.908942 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908871    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container"
Jul 16 22:23:21.908942 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908880    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container"
Jul 16 22:23:21.908942 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908895    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container"
Jul 16 22:23:21.908942 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908904    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container"
Jul 16 22:23:21.908942 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908932    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kube-rbac-proxy"
Jul 16 22:23:21.908942 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.908942    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kube-rbac-proxy"
Jul 16 22:23:21.909215 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.909052    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kube-rbac-proxy"
Jul 16 22:23:21.909215 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.909079    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6de4c344-fd98-4d3d-a0d8-04cdd26a394a" containerName="kserve-container"
Jul 16 22:23:21.909215 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.909090    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kube-rbac-proxy"
Jul 16 22:23:21.909215 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.909122    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6591ac58-f4a1-4d2b-ab1e-653071b8b3cd" containerName="kserve-container"
Jul 16 22:23:21.913447 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.913419    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:21.915630 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.915598    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-76a97-kube-rbac-proxy-sar-config\""
Jul 16 22:23:21.915756 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.915616    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"success-200-isvc-76a97-predictor-serving-cert\""
Jul 16 22:23:21.925705 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.925673    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"]
Jul 16 22:23:21.969472 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.969438    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:21.975451 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.975420    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"]
Jul 16 22:23:21.975674 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.975645    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container" containerID="cri-o://f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5" gracePeriod=30
Jul 16 22:23:21.975963 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:21.975940    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kube-rbac-proxy" containerID="cri-o://1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a" gracePeriod=30
Jul 16 22:23:22.029936 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.029899    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"]
Jul 16 22:23:22.033835 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.033805    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.040188 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.040164    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-76a97-predictor-serving-cert\""
Jul 16 22:23:22.040344 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.040186    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kserve-ci-e2e-test\"/\"error-404-isvc-76a97-kube-rbac-proxy-sar-config\""
Jul 16 22:23:22.045159 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.045091    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.045326 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.045203    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tctjx\" (UniqueName: \"kubernetes.io/projected/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-api-access-tctjx\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.045326 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.045287    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94458687-2e60-41ce-ab4e-30e3412d94d2-proxy-tls\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.046877 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.046848    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"]
Jul 16 22:23:22.067239 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.067200    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:22.145997 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.145963    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tctjx\" (UniqueName: \"kubernetes.io/projected/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-api-access-tctjx\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.146223 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.146062    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94458687-2e60-41ce-ab4e-30e3412d94d2-proxy-tls\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.146223 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.146160    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smvvz\" (UniqueName: \"kubernetes.io/projected/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-api-access-smvvz\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.146223 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.146197    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1c56da-d890-4369-be55-ce9b8cfee7b9-proxy-tls\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.146367 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.146222    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.146367 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.146288    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.146871 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.146848    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-rbac-proxy-sar-config\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.148571 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.148552    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94458687-2e60-41ce-ab4e-30e3412d94d2-proxy-tls\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.154663 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.154638    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tctjx\" (UniqueName: \"kubernetes.io/projected/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-api-access-tctjx\") pod \"success-200-isvc-76a97-predictor-7d9f876bfd-6nl99\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") " pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.197769 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.197733    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:22.226838 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.226740    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:22.247251 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.247219    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smvvz\" (UniqueName: \"kubernetes.io/projected/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-api-access-smvvz\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.247415 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.247260    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1c56da-d890-4369-be55-ce9b8cfee7b9-proxy-tls\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.247415 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.247281    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.247998 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.247978    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-rbac-proxy-sar-config\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.250345 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.250310    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1c56da-d890-4369-be55-ce9b8cfee7b9-proxy-tls\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.256496 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.256467    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-smvvz\" (UniqueName: \"kubernetes.io/projected/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-api-access-smvvz\") pod \"error-404-isvc-76a97-predictor-f65b59759-wj9k2\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") " pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.300296 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.300259    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:22.345679 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.345645    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:22.379422 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.379388    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"]
Jul 16 22:23:22.381584 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:23:22.381551    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94458687_2e60_41ce_ab4e_30e3412d94d2.slice/crio-034755c4be27b2c9a1af88720b9d357652c5a9abb38eaa58680e874f58bad253 WatchSource:0}: Error finding container 034755c4be27b2c9a1af88720b9d357652c5a9abb38eaa58680e874f58bad253: Status 404 returned error can't find the container with id 034755c4be27b2c9a1af88720b9d357652c5a9abb38eaa58680e874f58bad253
Jul 16 22:23:22.435715 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.435680    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" event={"ID":"94458687-2e60-41ce-ab4e-30e3412d94d2","Type":"ContainerStarted","Data":"034755c4be27b2c9a1af88720b9d357652c5a9abb38eaa58680e874f58bad253"}
Jul 16 22:23:22.437909 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.437881    2585 generic.go:356] "Generic (PLEG): container finished" podID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerID="df30702306dec8328da9d82e2b4a2dc9b4675041daa91d86cab90372e576ad9e" exitCode=2
Jul 16 22:23:22.438060 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.437962    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" event={"ID":"6a885f6f-87ef-4ca3-8830-d272bb0a0127","Type":"ContainerDied","Data":"df30702306dec8328da9d82e2b4a2dc9b4675041daa91d86cab90372e576ad9e"}
Jul 16 22:23:22.440344 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.440313    2585 generic.go:356] "Generic (PLEG): container finished" podID="653062fd-7a45-4f57-a939-69400a3be794" containerID="1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a" exitCode=2
Jul 16 22:23:22.440463 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.440361    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" event={"ID":"653062fd-7a45-4f57-a939-69400a3be794","Type":"ContainerDied","Data":"1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a"}
Jul 16 22:23:22.504832 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:22.504805    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"]
Jul 16 22:23:23.448051 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.448005    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" event={"ID":"3c1c56da-d890-4369-be55-ce9b8cfee7b9","Type":"ContainerStarted","Data":"8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94"}
Jul 16 22:23:23.448599 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.448082    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" event={"ID":"3c1c56da-d890-4369-be55-ce9b8cfee7b9","Type":"ContainerStarted","Data":"6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967"}
Jul 16 22:23:23.448599 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.448243    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:23.448599 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.448317    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" event={"ID":"3c1c56da-d890-4369-be55-ce9b8cfee7b9","Type":"ContainerStarted","Data":"fbb04794f0c68499f160a7406dd9cc62f80edb26af73cc183377de5ffa6c9e05"}
Jul 16 22:23:23.448599 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.448364    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:23.450164 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.450126    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" event={"ID":"94458687-2e60-41ce-ab4e-30e3412d94d2","Type":"ContainerStarted","Data":"186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411"}
Jul 16 22:23:23.450288 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.450156    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 16 22:23:23.450288 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.450192    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:23.450288 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.450213    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" event={"ID":"94458687-2e60-41ce-ab4e-30e3412d94d2","Type":"ContainerStarted","Data":"f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f"}
Jul 16 22:23:23.468907 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.468853    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podStartSLOduration=1.468834891 podStartE2EDuration="1.468834891s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:23:23.466234147 +0000 UTC m=+2003.270657477" watchObservedRunningTime="2026-07-16 22:23:23.468834891 +0000 UTC m=+2003.273258217"
Jul 16 22:23:23.489985 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.489921    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podStartSLOduration=2.489901373 podStartE2EDuration="2.489901373s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-16 22:23:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:23:23.487331699 +0000 UTC m=+2003.291755037" watchObservedRunningTime="2026-07-16 22:23:23.489901373 +0000 UTC m=+2003.294324698"
Jul 16 22:23:23.969931 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:23.969884    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.54:8643/healthz\": dial tcp 10.134.0.54:8643: connect: connection refused"
Jul 16 22:23:24.456147 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:24.456092    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:24.456617 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:24.456144    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 16 22:23:24.457581 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:24.457554    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 16 22:23:24.975646 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:24.975596    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.53:8643/healthz\": dial tcp 10.134.0.53:8643: connect: connection refused"
Jul 16 22:23:25.197288 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:25.197245    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.55:8080: connect: connection refused"
Jul 16 22:23:25.460763 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:25.460723    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 16 22:23:25.955464 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:25.955434    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:23:26.087187 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.087145    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jxsrp\" (UniqueName: \"kubernetes.io/projected/653062fd-7a45-4f57-a939-69400a3be794-kube-api-access-jxsrp\") pod \"653062fd-7a45-4f57-a939-69400a3be794\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") "
Jul 16 22:23:26.087415 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.087228    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653062fd-7a45-4f57-a939-69400a3be794-kube-rbac-proxy-sar-config\") pod \"653062fd-7a45-4f57-a939-69400a3be794\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") "
Jul 16 22:23:26.087415 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.087293    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653062fd-7a45-4f57-a939-69400a3be794-proxy-tls\") pod \"653062fd-7a45-4f57-a939-69400a3be794\" (UID: \"653062fd-7a45-4f57-a939-69400a3be794\") "
Jul 16 22:23:26.087592 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.087564    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653062fd-7a45-4f57-a939-69400a3be794-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "653062fd-7a45-4f57-a939-69400a3be794" (UID: "653062fd-7a45-4f57-a939-69400a3be794"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:26.089475 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.089449    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/653062fd-7a45-4f57-a939-69400a3be794-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "653062fd-7a45-4f57-a939-69400a3be794" (UID: "653062fd-7a45-4f57-a939-69400a3be794"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:23:26.089573 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.089501    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653062fd-7a45-4f57-a939-69400a3be794-kube-api-access-jxsrp" (OuterVolumeSpecName: "kube-api-access-jxsrp") pod "653062fd-7a45-4f57-a939-69400a3be794" (UID: "653062fd-7a45-4f57-a939-69400a3be794"). InnerVolumeSpecName "kube-api-access-jxsrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:23:26.188569 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.188528    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/653062fd-7a45-4f57-a939-69400a3be794-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:23:26.188569 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.188563    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jxsrp\" (UniqueName: \"kubernetes.io/projected/653062fd-7a45-4f57-a939-69400a3be794-kube-api-access-jxsrp\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:23:26.188569 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.188575    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/653062fd-7a45-4f57-a939-69400a3be794-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:23:26.202214 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.202163    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.56:8080: connect: connection refused"
Jul 16 22:23:26.471125 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.471065    2585 generic.go:356] "Generic (PLEG): container finished" podID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerID="d64ce63f1ffcc25a8b0d2edf11828f8ebbce4a0da47b68bdaf68574664d536dd" exitCode=0
Jul 16 22:23:26.471538 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.471133    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" event={"ID":"6a885f6f-87ef-4ca3-8830-d272bb0a0127","Type":"ContainerDied","Data":"d64ce63f1ffcc25a8b0d2edf11828f8ebbce4a0da47b68bdaf68574664d536dd"}
Jul 16 22:23:26.472933 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.472898    2585 generic.go:356] "Generic (PLEG): container finished" podID="653062fd-7a45-4f57-a939-69400a3be794" containerID="f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5" exitCode=0
Jul 16 22:23:26.473090 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.472936    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" event={"ID":"653062fd-7a45-4f57-a939-69400a3be794","Type":"ContainerDied","Data":"f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5"}
Jul 16 22:23:26.473090 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.472991    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq" event={"ID":"653062fd-7a45-4f57-a939-69400a3be794","Type":"ContainerDied","Data":"a4a02cb4feaf67d2404d7bac6da6e45a5a07a467004429c722e1cfe4bd48db50"}
Jul 16 22:23:26.473090 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.473008    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"
Jul 16 22:23:26.473090 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.473021    2585 scope.go:117] "RemoveContainer" containerID="1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a"
Jul 16 22:23:26.482310 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.482190    2585 scope.go:117] "RemoveContainer" containerID="f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5"
Jul 16 22:23:26.491593 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.491571    2585 scope.go:117] "RemoveContainer" containerID="1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a"
Jul 16 22:23:26.491924 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:23:26.491899    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a\": container with ID starting with 1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a not found: ID does not exist" containerID="1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a"
Jul 16 22:23:26.492019 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.491940    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a"} err="failed to get container status \"1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a\": rpc error: code = NotFound desc = could not find container \"1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a\": container with ID starting with 1264bdab10c6418f78b1edcbb8d3909ed324a3b96cb14c1d08fe4d1affdddf8a not found: ID does not exist"
Jul 16 22:23:26.492019 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.491966    2585 scope.go:117] "RemoveContainer" containerID="f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5"
Jul 16 22:23:26.492284 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:23:26.492261    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5\": container with ID starting with f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5 not found: ID does not exist" containerID="f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5"
Jul 16 22:23:26.492325 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.492296    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5"} err="failed to get container status \"f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5\": rpc error: code = NotFound desc = could not find container \"f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5\": container with ID starting with f757c7802098d8fd4b43dbdb3854fd8183c42939b034b82447dac43f0ca83bf5 not found: ID does not exist"
Jul 16 22:23:26.498346 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.498319    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"]
Jul 16 22:23:26.502115 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.502076    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:23:26.502775 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.502753    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-a3d07-predictor-f5d955f89-cthdq"]
Jul 16 22:23:26.590716 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.590678    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-rbac-proxy-sar-config\") pod \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") "
Jul 16 22:23:26.590890 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.590741    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a885f6f-87ef-4ca3-8830-d272bb0a0127-proxy-tls\") pod \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") "
Jul 16 22:23:26.590890 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.590781    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c72fb\" (UniqueName: \"kubernetes.io/projected/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-api-access-c72fb\") pod \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\" (UID: \"6a885f6f-87ef-4ca3-8830-d272bb0a0127\") "
Jul 16 22:23:26.591071 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.591047    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "6a885f6f-87ef-4ca3-8830-d272bb0a0127" (UID: "6a885f6f-87ef-4ca3-8830-d272bb0a0127"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:23:26.591183 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.591165    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:23:26.592941 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.592914    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a885f6f-87ef-4ca3-8830-d272bb0a0127-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "6a885f6f-87ef-4ca3-8830-d272bb0a0127" (UID: "6a885f6f-87ef-4ca3-8830-d272bb0a0127"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:23:26.593050 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.592951    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-api-access-c72fb" (OuterVolumeSpecName: "kube-api-access-c72fb") pod "6a885f6f-87ef-4ca3-8830-d272bb0a0127" (UID: "6a885f6f-87ef-4ca3-8830-d272bb0a0127"). InnerVolumeSpecName "kube-api-access-c72fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:23:26.691763 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.691643    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/6a885f6f-87ef-4ca3-8830-d272bb0a0127-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:23:26.691763 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.691697    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c72fb\" (UniqueName: \"kubernetes.io/projected/6a885f6f-87ef-4ca3-8830-d272bb0a0127-kube-api-access-c72fb\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:23:26.827498 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:26.827461    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="653062fd-7a45-4f57-a939-69400a3be794" path="/var/lib/kubelet/pods/653062fd-7a45-4f57-a939-69400a3be794/volumes"
Jul 16 22:23:26.877570 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:23:26.877532    2585 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a885f6f_87ef_4ca3_8830_d272bb0a0127.slice/crio-3e40afc2cb3d8313eac52ff264dd4513db410e9351ba9f05bfde26a5925b2dab\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a885f6f_87ef_4ca3_8830_d272bb0a0127.slice\": RecentStats: unable to find data in memory cache]"
Jul 16 22:23:27.481334 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:27.481299    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"
Jul 16 22:23:27.481334 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:27.481310    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9" event={"ID":"6a885f6f-87ef-4ca3-8830-d272bb0a0127","Type":"ContainerDied","Data":"3e40afc2cb3d8313eac52ff264dd4513db410e9351ba9f05bfde26a5925b2dab"}
Jul 16 22:23:27.481881 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:27.481425    2585 scope.go:117] "RemoveContainer" containerID="df30702306dec8328da9d82e2b4a2dc9b4675041daa91d86cab90372e576ad9e"
Jul 16 22:23:27.490599 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:27.490574    2585 scope.go:117] "RemoveContainer" containerID="d64ce63f1ffcc25a8b0d2edf11828f8ebbce4a0da47b68bdaf68574664d536dd"
Jul 16 22:23:27.506696 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:27.506656    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"]
Jul 16 22:23:27.511577 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:27.511531    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-a3d07-predictor-546c97cfb4-9wvg9"]
Jul 16 22:23:28.827419 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:28.827384    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" path="/var/lib/kubelet/pods/6a885f6f-87ef-4ca3-8830-d272bb0a0127/volumes"
Jul 16 22:23:29.461447 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:29.461414    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:23:29.462093 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:29.462057    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 16 22:23:30.466339 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:30.466307    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:23:30.467014 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:30.466982    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 16 22:23:35.198407 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:35.198365    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:23:36.203141 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:36.203089    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:23:39.462864 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:39.462819    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 16 22:23:40.467462 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:40.467419    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 16 22:23:47.516637 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:47.516593    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:47.941291 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:47.941197    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:23:49.462579 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:49.462538    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 16 22:23:50.467243 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:50.467196    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 16 22:23:59.462143 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:23:59.462080    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 16 22:24:00.467311 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:24:00.467263    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 16 22:24:09.462944 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:24:09.462898    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 16 22:24:10.468827 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:24:10.468788    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:24:19.463811 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:24:19.463778    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:24:21.974282 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:24:21.974247    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:24:22.513408 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:24:22.513322    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:25:00.853685 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:25:00.853637    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:25:00.855570 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:25:00.855546    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:26:07.009017 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:26:07.008922    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:26:25.651296 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:26:25.651253    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:27:12.689695 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:27:12.689653    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:30:00.888452 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:30:00.888417    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:30:00.891933 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:30:00.891912    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:30:15.890796 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:30:15.890754    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:30:17.879259 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:30:17.879217    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:30:28.692326 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:30:28.692238    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:36.777209 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.777156    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:36.859515 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.859473    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"]
Jul 16 22:32:36.859825 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.859795    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container" containerID="cri-o://f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f" gracePeriod=30
Jul 16 22:32:36.859947 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.859844    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kube-rbac-proxy" containerID="cri-o://186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411" gracePeriod=30
Jul 16 22:32:36.892905 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.892866    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:36.903129 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.903074    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"]
Jul 16 22:32:36.903370 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.903339    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" containerID="cri-o://6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967" gracePeriod=30
Jul 16 22:32:36.903483 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.903352    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kube-rbac-proxy" containerID="cri-o://8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94" gracePeriod=30
Jul 16 22:32:36.963950 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:36.963902    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:32:37.386182 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:37.386142    2585 generic.go:356] "Generic (PLEG): container finished" podID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerID="8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94" exitCode=2
Jul 16 22:32:37.386389 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:37.386214    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" event={"ID":"3c1c56da-d890-4369-be55-ce9b8cfee7b9","Type":"ContainerDied","Data":"8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94"}
Jul 16 22:32:37.388017 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:37.387994    2585 generic.go:356] "Generic (PLEG): container finished" podID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerID="186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411" exitCode=2
Jul 16 22:32:37.388137 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:37.388051    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" event={"ID":"94458687-2e60-41ce-ab4e-30e3412d94d2","Type":"ContainerDied","Data":"186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411"}
Jul 16 22:32:39.457257 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:39.457210    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.58:8643/healthz\": dial tcp 10.134.0.58:8643: connect: connection refused"
Jul 16 22:32:39.462487 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:39.462456    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.58:8080: connect: connection refused"
Jul 16 22:32:40.461868 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.461827    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.57:8643/healthz\": dial tcp 10.134.0.57:8643: connect: connection refused"
Jul 16 22:32:40.467366 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.467328    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container" probeResult="failure" output="dial tcp 10.134.0.57:8080: connect: connection refused"
Jul 16 22:32:40.680316 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.680248    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:32:40.772418 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.772380    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tctjx\" (UniqueName: \"kubernetes.io/projected/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-api-access-tctjx\") pod \"94458687-2e60-41ce-ab4e-30e3412d94d2\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") "
Jul 16 22:32:40.772637 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.772440    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94458687-2e60-41ce-ab4e-30e3412d94d2-proxy-tls\") pod \"94458687-2e60-41ce-ab4e-30e3412d94d2\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") "
Jul 16 22:32:40.772637 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.772512    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-rbac-proxy-sar-config\") pod \"94458687-2e60-41ce-ab4e-30e3412d94d2\" (UID: \"94458687-2e60-41ce-ab4e-30e3412d94d2\") "
Jul 16 22:32:40.772912 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.772877    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "94458687-2e60-41ce-ab4e-30e3412d94d2" (UID: "94458687-2e60-41ce-ab4e-30e3412d94d2"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:40.774707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.774679    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94458687-2e60-41ce-ab4e-30e3412d94d2-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "94458687-2e60-41ce-ab4e-30e3412d94d2" (UID: "94458687-2e60-41ce-ab4e-30e3412d94d2"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:32:40.774812 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.774730    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-api-access-tctjx" (OuterVolumeSpecName: "kube-api-access-tctjx") pod "94458687-2e60-41ce-ab4e-30e3412d94d2" (UID: "94458687-2e60-41ce-ab4e-30e3412d94d2"). InnerVolumeSpecName "kube-api-access-tctjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:32:40.856706 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.856679    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:32:40.873961 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.873926    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1c56da-d890-4369-be55-ce9b8cfee7b9-proxy-tls\") pod \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") "
Jul 16 22:32:40.874131 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.873974    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-smvvz\" (UniqueName: \"kubernetes.io/projected/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-api-access-smvvz\") pod \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") "
Jul 16 22:32:40.874131 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.874044    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-rbac-proxy-sar-config\") pod \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\" (UID: \"3c1c56da-d890-4369-be55-ce9b8cfee7b9\") "
Jul 16 22:32:40.874346 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.874324    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:32:40.874422 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.874351    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tctjx\" (UniqueName: \"kubernetes.io/projected/94458687-2e60-41ce-ab4e-30e3412d94d2-kube-api-access-tctjx\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:32:40.874422 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.874367    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/94458687-2e60-41ce-ab4e-30e3412d94d2-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:32:40.874422 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.874361    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3c1c56da-d890-4369-be55-ce9b8cfee7b9" (UID: "3c1c56da-d890-4369-be55-ce9b8cfee7b9"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:32:40.876529 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.876495    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c1c56da-d890-4369-be55-ce9b8cfee7b9-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3c1c56da-d890-4369-be55-ce9b8cfee7b9" (UID: "3c1c56da-d890-4369-be55-ce9b8cfee7b9"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:32:40.876636 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.876547    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-api-access-smvvz" (OuterVolumeSpecName: "kube-api-access-smvvz") pod "3c1c56da-d890-4369-be55-ce9b8cfee7b9" (UID: "3c1c56da-d890-4369-be55-ce9b8cfee7b9"). InnerVolumeSpecName "kube-api-access-smvvz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:32:40.975388 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.975294    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3c1c56da-d890-4369-be55-ce9b8cfee7b9-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:32:40.975388 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.975326    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smvvz\" (UniqueName: \"kubernetes.io/projected/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-api-access-smvvz\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:32:40.975388 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:40.975338    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3c1c56da-d890-4369-be55-ce9b8cfee7b9-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:32:41.416402 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.416370    2585 generic.go:356] "Generic (PLEG): container finished" podID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerID="f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f" exitCode=0
Jul 16 22:32:41.416601 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.416449    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"
Jul 16 22:32:41.416601 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.416460    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" event={"ID":"94458687-2e60-41ce-ab4e-30e3412d94d2","Type":"ContainerDied","Data":"f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f"}
Jul 16 22:32:41.416601 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.416548    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99" event={"ID":"94458687-2e60-41ce-ab4e-30e3412d94d2","Type":"ContainerDied","Data":"034755c4be27b2c9a1af88720b9d357652c5a9abb38eaa58680e874f58bad253"}
Jul 16 22:32:41.416601 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.416586    2585 scope.go:117] "RemoveContainer" containerID="186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411"
Jul 16 22:32:41.418254 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.418230    2585 generic.go:356] "Generic (PLEG): container finished" podID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerID="6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967" exitCode=0
Jul 16 22:32:41.418390 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.418270    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" event={"ID":"3c1c56da-d890-4369-be55-ce9b8cfee7b9","Type":"ContainerDied","Data":"6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967"}
Jul 16 22:32:41.418390 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.418311    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2" event={"ID":"3c1c56da-d890-4369-be55-ce9b8cfee7b9","Type":"ContainerDied","Data":"fbb04794f0c68499f160a7406dd9cc62f80edb26af73cc183377de5ffa6c9e05"}
Jul 16 22:32:41.418390 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.418388    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"
Jul 16 22:32:41.434945 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.434923    2585 scope.go:117] "RemoveContainer" containerID="f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f"
Jul 16 22:32:41.443833 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.443804    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"]
Jul 16 22:32:41.444015 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.444000    2585 scope.go:117] "RemoveContainer" containerID="186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411"
Jul 16 22:32:41.444370 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:32:41.444344    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411\": container with ID starting with 186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411 not found: ID does not exist" containerID="186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411"
Jul 16 22:32:41.444475 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.444383    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411"} err="failed to get container status \"186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411\": rpc error: code = NotFound desc = could not find container \"186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411\": container with ID starting with 186cae67def77b0756b269954e327721be6a90bebf4d824a4329d89a4e8f1411 not found: ID does not exist"
Jul 16 22:32:41.444475 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.444410    2585 scope.go:117] "RemoveContainer" containerID="f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f"
Jul 16 22:32:41.444695 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:32:41.444675    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f\": container with ID starting with f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f not found: ID does not exist" containerID="f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f"
Jul 16 22:32:41.444771 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.444703    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f"} err="failed to get container status \"f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f\": rpc error: code = NotFound desc = could not find container \"f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f\": container with ID starting with f72d645688cfb959c58c8f8d6d97182a51d0be936a55a57a699b57052bbff73f not found: ID does not exist"
Jul 16 22:32:41.444771 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.444720    2585 scope.go:117] "RemoveContainer" containerID="8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94"
Jul 16 22:32:41.447959 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.447933    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-76a97-predictor-7d9f876bfd-6nl99"]
Jul 16 22:32:41.453561 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.453541    2585 scope.go:117] "RemoveContainer" containerID="6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967"
Jul 16 22:32:41.460078 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.460030    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"]
Jul 16 22:32:41.463439 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.463415    2585 scope.go:117] "RemoveContainer" containerID="8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94"
Jul 16 22:32:41.464025 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:32:41.464002    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94\": container with ID starting with 8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94 not found: ID does not exist" containerID="8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94"
Jul 16 22:32:41.464090 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.464035    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94"} err="failed to get container status \"8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94\": rpc error: code = NotFound desc = could not find container \"8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94\": container with ID starting with 8e79c672ae6c3873ec5b4e6ce09cf6d73dc1f366f69bd917848ebbc59a854b94 not found: ID does not exist"
Jul 16 22:32:41.464090 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.464055    2585 scope.go:117] "RemoveContainer" containerID="6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967"
Jul 16 22:32:41.464417 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:32:41.464390    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967\": container with ID starting with 6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967 not found: ID does not exist" containerID="6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967"
Jul 16 22:32:41.464486 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.464427    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967"} err="failed to get container status \"6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967\": rpc error: code = NotFound desc = could not find container \"6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967\": container with ID starting with 6c7236c8126ca514e7465284cd092496062d4c25d8e22ee385360bf5eab37967 not found: ID does not exist"
Jul 16 22:32:41.465579 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:41.465559    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-76a97-predictor-f65b59759-wj9k2"]
Jul 16 22:32:42.827214 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:42.827178    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" path="/var/lib/kubelet/pods/3c1c56da-d890-4369-be55-ce9b8cfee7b9/volumes"
Jul 16 22:32:42.827707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:32:42.827691    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" path="/var/lib/kubelet/pods/94458687-2e60-41ce-ab4e-30e3412d94d2/volumes"
Jul 16 22:33:58.526330 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:33:58.526285    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:35:00.920974 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:35:00.920856    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:35:00.925028 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:35:00.924755    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:39:22.897896 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:39:22.897788    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:00.955884 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:00.955766    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:40:00.961642 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:00.961618    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:40:06.955445 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:06.955395    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:07.070649 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.070614    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"]
Jul 16 22:40:07.070863 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.070838    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container" containerID="cri-o://4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290" gracePeriod=30
Jul 16 22:40:07.070959 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.070929    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kube-rbac-proxy" containerID="cri-o://b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3" gracePeriod=30
Jul 16 22:40:07.102454 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.102421    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:07.139040 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.138999    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"]
Jul 16 22:40:07.139277 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.139247    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container" containerID="cri-o://cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7" gracePeriod=30
Jul 16 22:40:07.139416 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.139301    2585 kuberuntime_container.go:873] "Killing container with a grace period" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kube-rbac-proxy" containerID="cri-o://cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe" gracePeriod=30
Jul 16 22:40:07.197808 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.197765    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:07.683781 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.683678    2585 generic.go:356] "Generic (PLEG): container finished" podID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerID="b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3" exitCode=2
Jul 16 22:40:07.683781 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.683749    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" event={"ID":"97b91099-75f0-47a7-994a-fba7ebdfd400","Type":"ContainerDied","Data":"b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3"}
Jul 16 22:40:07.685753 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.685727    2585 generic.go:356] "Generic (PLEG): container finished" podID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerID="cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe" exitCode=2
Jul 16 22:40:07.685885 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:07.685782    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" event={"ID":"3e2c3c09-dd80-4362-9c24-6b7bb097e018","Type":"ContainerDied","Data":"cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe"}
Jul 16 22:40:08.388348 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:08.388313    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:08.512474 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:08.512418    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:10.191506 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:10.191464    2585 prober.go:120] "Probe failed" probeType="Readiness" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kube-rbac-proxy" probeResult="failure" output="Get \"https://10.134.0.55:8643/healthz\": dial tcp 10.134.0.55:8643: connect: connection refused"
Jul 16 22:40:10.957852 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:10.957823    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:40:10.965416 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:10.965343    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:40:11.051940 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.051907    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b91099-75f0-47a7-994a-fba7ebdfd400-proxy-tls\") pod \"97b91099-75f0-47a7-994a-fba7ebdfd400\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") "
Jul 16 22:40:11.052152 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.051981    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k6zp7\" (UniqueName: \"kubernetes.io/projected/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-api-access-k6zp7\") pod \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") "
Jul 16 22:40:11.052152 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.052011    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls\") pod \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") "
Jul 16 22:40:11.052152 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.052060    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-rbac-proxy-sar-config\") pod \"97b91099-75f0-47a7-994a-fba7ebdfd400\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") "
Jul 16 22:40:11.052339 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.052236    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6dgs4\" (UniqueName: \"kubernetes.io/projected/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-api-access-6dgs4\") pod \"97b91099-75f0-47a7-994a-fba7ebdfd400\" (UID: \"97b91099-75f0-47a7-994a-fba7ebdfd400\") "
Jul 16 22:40:11.052339 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.052290    2585 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-rbac-proxy-sar-config\") pod \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\" (UID: \"3e2c3c09-dd80-4362-9c24-6b7bb097e018\") "
Jul 16 22:40:11.052456 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.052431    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "97b91099-75f0-47a7-994a-fba7ebdfd400" (UID: "97b91099-75f0-47a7-994a-fba7ebdfd400"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:11.052684 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.052649    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:40:11.052807 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.052727    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-rbac-proxy-sar-config" (OuterVolumeSpecName: "kube-rbac-proxy-sar-config") pod "3e2c3c09-dd80-4362-9c24-6b7bb097e018" (UID: "3e2c3c09-dd80-4362-9c24-6b7bb097e018"). InnerVolumeSpecName "kube-rbac-proxy-sar-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 16 22:40:11.054357 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.054329    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-api-access-k6zp7" (OuterVolumeSpecName: "kube-api-access-k6zp7") pod "3e2c3c09-dd80-4362-9c24-6b7bb097e018" (UID: "3e2c3c09-dd80-4362-9c24-6b7bb097e018"). InnerVolumeSpecName "kube-api-access-k6zp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:11.054539 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.054518    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97b91099-75f0-47a7-994a-fba7ebdfd400-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "97b91099-75f0-47a7-994a-fba7ebdfd400" (UID: "97b91099-75f0-47a7-994a-fba7ebdfd400"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:40:11.054722 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.054700    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-api-access-6dgs4" (OuterVolumeSpecName: "kube-api-access-6dgs4") pod "97b91099-75f0-47a7-994a-fba7ebdfd400" (UID: "97b91099-75f0-47a7-994a-fba7ebdfd400"). InnerVolumeSpecName "kube-api-access-6dgs4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 16 22:40:11.054774 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.054709    2585 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls" (OuterVolumeSpecName: "proxy-tls") pod "3e2c3c09-dd80-4362-9c24-6b7bb097e018" (UID: "3e2c3c09-dd80-4362-9c24-6b7bb097e018"). InnerVolumeSpecName "proxy-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 16 22:40:11.153536 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.153495    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6zp7\" (UniqueName: \"kubernetes.io/projected/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-api-access-k6zp7\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:40:11.153536 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.153530    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/3e2c3c09-dd80-4362-9c24-6b7bb097e018-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:40:11.153536 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.153539    2585 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6dgs4\" (UniqueName: \"kubernetes.io/projected/97b91099-75f0-47a7-994a-fba7ebdfd400-kube-api-access-6dgs4\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:40:11.153790 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.153549    2585 reconciler_common.go:299] "Volume detached for volume \"kube-rbac-proxy-sar-config\" (UniqueName: \"kubernetes.io/configmap/3e2c3c09-dd80-4362-9c24-6b7bb097e018-kube-rbac-proxy-sar-config\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:40:11.153790 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.153561    2585 reconciler_common.go:299] "Volume detached for volume \"proxy-tls\" (UniqueName: \"kubernetes.io/secret/97b91099-75f0-47a7-994a-fba7ebdfd400-proxy-tls\") on node \"ip-10-0-129-202.ec2.internal\" DevicePath \"\""
Jul 16 22:40:11.708708 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.708657    2585 generic.go:356] "Generic (PLEG): container finished" podID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerID="4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290" exitCode=0
Jul 16 22:40:11.709245 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.708746    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"
Jul 16 22:40:11.709245 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.708746    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" event={"ID":"97b91099-75f0-47a7-994a-fba7ebdfd400","Type":"ContainerDied","Data":"4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290"}
Jul 16 22:40:11.709245 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.708870    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h" event={"ID":"97b91099-75f0-47a7-994a-fba7ebdfd400","Type":"ContainerDied","Data":"f603ed17924dc3248bbd3dd5abed336442777134087ed6d606ac13f0cb31c4ba"}
Jul 16 22:40:11.709245 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.708910    2585 scope.go:117] "RemoveContainer" containerID="b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3"
Jul 16 22:40:11.711007 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.710978    2585 generic.go:356] "Generic (PLEG): container finished" podID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerID="cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7" exitCode=0
Jul 16 22:40:11.711142 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.711018    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" event={"ID":"3e2c3c09-dd80-4362-9c24-6b7bb097e018","Type":"ContainerDied","Data":"cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7"}
Jul 16 22:40:11.711142 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.711049    2585 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"
Jul 16 22:40:11.711142 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.711089    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7" event={"ID":"3e2c3c09-dd80-4362-9c24-6b7bb097e018","Type":"ContainerDied","Data":"7b2fec4f683fa20d11ce3d7c72f8ab2da055db6fd0fcfb8c79453e5a30843264"}
Jul 16 22:40:11.724130 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.724109    2585 scope.go:117] "RemoveContainer" containerID="4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290"
Jul 16 22:40:11.732543 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.732518    2585 scope.go:117] "RemoveContainer" containerID="b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3"
Jul 16 22:40:11.732874 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:40:11.732847    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3\": container with ID starting with b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3 not found: ID does not exist" containerID="b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3"
Jul 16 22:40:11.732952 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.732883    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3"} err="failed to get container status \"b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3\": rpc error: code = NotFound desc = could not find container \"b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3\": container with ID starting with b0681eb2434eb4c0828846ca93f6490c19a29e394de7c553f6cf7e1f356447f3 not found: ID does not exist"
Jul 16 22:40:11.732952 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.732901    2585 scope.go:117] "RemoveContainer" containerID="4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290"
Jul 16 22:40:11.733198 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:40:11.733182    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290\": container with ID starting with 4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290 not found: ID does not exist" containerID="4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290"
Jul 16 22:40:11.733250 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.733203    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290"} err="failed to get container status \"4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290\": rpc error: code = NotFound desc = could not find container \"4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290\": container with ID starting with 4b7f9dffa7c2819046a93c796ad71b44a79da2599c08c8a872006da77d957290 not found: ID does not exist"
Jul 16 22:40:11.733250 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.733217    2585 scope.go:117] "RemoveContainer" containerID="cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe"
Jul 16 22:40:11.741226 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.741194    2585 scope.go:117] "RemoveContainer" containerID="cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7"
Jul 16 22:40:11.752501 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.752165    2585 scope.go:117] "RemoveContainer" containerID="cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe"
Jul 16 22:40:11.752501 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.752219    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"]
Jul 16 22:40:11.752963 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:40:11.752943    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe\": container with ID starting with cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe not found: ID does not exist" containerID="cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe"
Jul 16 22:40:11.753050 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.752973    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe"} err="failed to get container status \"cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe\": rpc error: code = NotFound desc = could not find container \"cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe\": container with ID starting with cbcd26c164ae713e68e6d47087421cab53181b05b53d7d25f481a59a1b150cfe not found: ID does not exist"
Jul 16 22:40:11.753050 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.752999    2585 scope.go:117] "RemoveContainer" containerID="cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7"
Jul 16 22:40:11.753579 ip-10-0-129-202 kubenswrapper[2585]: E0716 22:40:11.753552    2585 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7\": container with ID starting with cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7 not found: ID does not exist" containerID="cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7"
Jul 16 22:40:11.753677 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.753585    2585 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7"} err="failed to get container status \"cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7\": rpc error: code = NotFound desc = could not find container \"cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7\": container with ID starting with cae1d8761c8c846f50fdfe3364682e8c2638ffffa2828e51c4384173f09279c7 not found: ID does not exist"
Jul 16 22:40:11.763352 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.763322    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/success-200-isvc-93462-predictor-759cb758b6-4cnb7"]
Jul 16 22:40:11.786260 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.786226    2585 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"]
Jul 16 22:40:11.790714 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:11.790672    2585 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kserve-ci-e2e-test/error-404-isvc-93462-predictor-655689f48c-4gb5h"]
Jul 16 22:40:12.827961 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:12.827923    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" path="/var/lib/kubelet/pods/3e2c3c09-dd80-4362-9c24-6b7bb097e018/volumes"
Jul 16 22:40:12.828420 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:12.828404    2585 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" path="/var/lib/kubelet/pods/97b91099-75f0-47a7-994a-fba7ebdfd400/volumes"
Jul 16 22:40:22.546541 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:22.546477    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:40.054407 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:40.054369    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:40.243335 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:40.243262    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:40.410423 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:40.410335    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:43.737852 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:43.737813    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-5wx6j_00df8ce7-3aa0-4336-a467-766ba77730b8/global-pull-secret-syncer/0.log"
Jul 16 22:40:43.883703 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:43.883656    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log"
Jul 16 22:40:44.024585 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:44.024486    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-202.ec2.internal_a549896eb4d5c414c32f605c3e5efd2f/haproxy/0.log"
Jul 16 22:40:45.943409 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:45.943370    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:46.024149 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:46.024087    2585 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-cms5w_e8ccc6eb-91cf-4474-bdbd-f8c89506deb7/konnectivity-agent/0.log" limit=1024
Jul 16 22:40:48.060721 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.060686    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1dc36b25-f6be-4d30-8669-63324253467c/alertmanager/0.log"
Jul 16 22:40:48.106973 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.106932    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1dc36b25-f6be-4d30-8669-63324253467c/config-reloader/0.log"
Jul 16 22:40:48.141368 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.141336    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1dc36b25-f6be-4d30-8669-63324253467c/kube-rbac-proxy-web/0.log"
Jul 16 22:40:48.187612 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.187584    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1dc36b25-f6be-4d30-8669-63324253467c/kube-rbac-proxy/0.log"
Jul 16 22:40:48.226636 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.226605    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1dc36b25-f6be-4d30-8669-63324253467c/kube-rbac-proxy-metric/0.log"
Jul 16 22:40:48.259267 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.259233    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1dc36b25-f6be-4d30-8669-63324253467c/prom-label-proxy/0.log"
Jul 16 22:40:48.298260 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.298230    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_1dc36b25-f6be-4d30-8669-63324253467c/init-config-reloader/0.log"
Jul 16 22:40:48.352438 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.352351    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-76cf8966f8-grzb8_1defe86c-f9b6-4b89-b263-3f8d84c15935/cluster-monitoring-operator/0.log"
Jul 16 22:40:48.397858 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.397830    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-plpcr_c6fc99cf-c316-4484-a451-c51854b1727e/kube-state-metrics/0.log"
Jul 16 22:40:48.427319 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.427290    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-plpcr_c6fc99cf-c316-4484-a451-c51854b1727e/kube-rbac-proxy-main/0.log"
Jul 16 22:40:48.463503 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.463470    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-56c5594674-plpcr_c6fc99cf-c316-4484-a451-c51854b1727e/kube-rbac-proxy-self/0.log"
Jul 16 22:40:48.514419 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.514386    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-69bf49fd5f-s7lk5_1d729d56-c88f-49ff-8882-e4e28d58859a/metrics-server/0.log"
Jul 16 22:40:48.562355 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.562327    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_monitoring-plugin-7b7b7bc8bf-56b6b_9476ce3d-86fb-4e4e-bede-d721052135d4/monitoring-plugin/0.log"
Jul 16 22:40:48.846369 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.846339    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jvqs7_34db72e0-c3d3-443e-9b0b-d89ea20e4afb/node-exporter/0.log"
Jul 16 22:40:48.888270 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.888246    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jvqs7_34db72e0-c3d3-443e-9b0b-d89ea20e4afb/kube-rbac-proxy/0.log"
Jul 16 22:40:48.921950 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:48.921920    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-jvqs7_34db72e0-c3d3-443e-9b0b-d89ea20e4afb/init-textfile/0.log"
Jul 16 22:40:49.565677 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.565639    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-7pvwx_dead8117-40cc-4617-b2e7-560ec9ad992e/prometheus-operator/0.log"
Jul 16 22:40:49.611521 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.611485    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-5945b5cfd-7pvwx_dead8117-40cc-4617-b2e7-560ec9ad992e/kube-rbac-proxy/0.log"
Jul 16 22:40:49.662290 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.662260    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-85574b5f49-6d7cf_f7e248a8-1d54-411c-8966-a727b77419f1/prometheus-operator-admission-webhook/0.log"
Jul 16 22:40:49.708140 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.708087    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b87bbcfcf-b69xm_df481018-75a5-4b01-9797-ee333a342de0/telemeter-client/0.log"
Jul 16 22:40:49.752867 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.752617    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b87bbcfcf-b69xm_df481018-75a5-4b01-9797-ee333a342de0/reload/0.log"
Jul 16 22:40:49.800581 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.800549    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7b87bbcfcf-b69xm_df481018-75a5-4b01-9797-ee333a342de0/kube-rbac-proxy/0.log"
Jul 16 22:40:49.852477 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.852398    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86656b897-m6jt9_69e0a640-0edc-4cb6-add3-0e5a25953d8a/thanos-query/0.log"
Jul 16 22:40:49.891681 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.891646    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86656b897-m6jt9_69e0a640-0edc-4cb6-add3-0e5a25953d8a/kube-rbac-proxy-web/0.log"
Jul 16 22:40:49.937260 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.937231    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86656b897-m6jt9_69e0a640-0edc-4cb6-add3-0e5a25953d8a/kube-rbac-proxy/0.log"
Jul 16 22:40:49.970836 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:49.970799    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86656b897-m6jt9_69e0a640-0edc-4cb6-add3-0e5a25953d8a/prom-label-proxy/0.log"
Jul 16 22:40:50.000369 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:50.000341    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86656b897-m6jt9_69e0a640-0edc-4cb6-add3-0e5a25953d8a/kube-rbac-proxy-rules/0.log"
Jul 16 22:40:50.039518 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:50.039489    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-86656b897-m6jt9_69e0a640-0edc-4cb6-add3-0e5a25953d8a/kube-rbac-proxy-metrics/0.log"
Jul 16 22:40:51.307526 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.307484    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-7bc7489cb5-l262c_c9469eba-c73f-4135-aa80-f12cf16ee555/networking-console-plugin/0.log"
Jul 16 22:40:51.820770 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.820738    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/1.log"
Jul 16 22:40:51.827062 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.827033    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-565b7b569d-5jwjh_aa88b577-be62-48bb-bf4d-2eadb3b2d857/console-operator/2.log"
Jul 16 22:40:51.950301 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950253    2585 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"]
Jul 16 22:40:51.950799 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950776    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container"
Jul 16 22:40:51.950799 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950798    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950807    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950816    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950832    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950839    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950845    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950851    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950865    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950871    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950877    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950882    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950891    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950900    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950910    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950918    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950926    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950931    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950939    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950944    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950951    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950956    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950964    2585 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container"
Jul 16 22:40:51.951021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.950972    2585 state_mem.go:107] "Deleted CPUSet assignment" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951044    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kserve-container"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951055    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951066    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951074    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="653062fd-7a45-4f57-a939-69400a3be794" containerName="kserve-container"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951083    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kserve-container"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951090    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e2c3c09-dd80-4362-9c24-6b7bb097e018" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951109    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kserve-container"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951115    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kserve-container"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951123    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="94458687-2e60-41ce-ab4e-30e3412d94d2" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951130    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c1c56da-d890-4369-be55-ce9b8cfee7b9" containerName="kube-rbac-proxy"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951136    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a885f6f-87ef-4ca3-8830-d272bb0a0127" containerName="kserve-container"
Jul 16 22:40:51.951707 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.951142    2585 memory_manager.go:368] "RemoveStaleState removing state" podUID="97b91099-75f0-47a7-994a-fba7ebdfd400" containerName="kube-rbac-proxy"
Jul 16 22:40:51.954454 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.954435    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:51.957228 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.957203    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jnrjl\"/\"kube-root-ca.crt\""
Jul 16 22:40:51.957348 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.957280    2585 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jnrjl\"/\"openshift-service-ca.crt\""
Jul 16 22:40:51.958021 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.958005    2585 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jnrjl\"/\"default-dockercfg-gv8jf\""
Jul 16 22:40:51.968462 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:51.968427    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"]
Jul 16 22:40:52.133387 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.133292    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-lib-modules\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.133387 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.133341    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-podres\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.133592 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.133447    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-sys\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.133592 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.133496    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-proc\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.133592 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.133515    2585 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpq4d\" (UniqueName: \"kubernetes.io/projected/73812c3f-7fa9-4cc3-a81c-9a036da251da-kube-api-access-wpq4d\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234615 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234567    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-lib-modules\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234615 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234628    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-podres\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234872 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234672    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-sys\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234872 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234704    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-proc\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234872 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234721    2585 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpq4d\" (UniqueName: \"kubernetes.io/projected/73812c3f-7fa9-4cc3-a81c-9a036da251da-kube-api-access-wpq4d\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234872 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234790    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-sys\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234872 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234803    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-proc\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234872 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234792    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-lib-modules\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.234872 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.234847    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/73812c3f-7fa9-4cc3-a81c-9a036da251da-podres\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.244460 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.244424    2585 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpq4d\" (UniqueName: \"kubernetes.io/projected/73812c3f-7fa9-4cc3-a81c-9a036da251da-kube-api-access-wpq4d\") pod \"perf-node-gather-daemonset-hbx5f\" (UID: \"73812c3f-7fa9-4cc3-a81c-9a036da251da\") " pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.265427 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.265337    2585 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.330421 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.330379    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-7c479d9b44-bctf4_db081b6b-faa0-4113-ac3e-802dccb6d79d/console/0.log"
Jul 16 22:40:52.384351 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.384249    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-5bc67dcd46-dmksn_c4a90ba1-3a1e-42b3-91a4-287b04054e2d/download-server/0.log"
Jul 16 22:40:52.413931 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.413900    2585 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"]
Jul 16 22:40:52.415822 ip-10-0-129-202 kubenswrapper[2585]: W0716 22:40:52.415794    2585 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod73812c3f_7fa9_4cc3_a81c_9a036da251da.slice/crio-bc7114be32dfea4d5e0e896e7000aaa5e4f648be70ae06c7bcf6b7f3adbfa6b1 WatchSource:0}: Error finding container bc7114be32dfea4d5e0e896e7000aaa5e4f648be70ae06c7bcf6b7f3adbfa6b1: Status 404 returned error can't find the container with id bc7114be32dfea4d5e0e896e7000aaa5e4f648be70ae06c7bcf6b7f3adbfa6b1
Jul 16 22:40:52.920331 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.920289    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f" event={"ID":"73812c3f-7fa9-4cc3-a81c-9a036da251da","Type":"ContainerStarted","Data":"b8535f315f2e1095f3308d7c29f3adc03149169d09af40dd3befb2bec4668030"}
Jul 16 22:40:52.920543 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.920352    2585 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f" event={"ID":"73812c3f-7fa9-4cc3-a81c-9a036da251da","Type":"ContainerStarted","Data":"bc7114be32dfea4d5e0e896e7000aaa5e4f648be70ae06c7bcf6b7f3adbfa6b1"}
Jul 16 22:40:52.920543 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.920378    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:52.939412 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.939351    2585 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f" podStartSLOduration=1.93933126 podStartE2EDuration="1.93933126s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-16 22:40:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-16 22:40:52.936864056 +0000 UTC m=+3052.741287393" watchObservedRunningTime="2026-07-16 22:40:52.93933126 +0000 UTC m=+3052.743754637"
Jul 16 22:40:52.984951 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:52.984918    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78cf4c6f6-jttvd_e8f0d57d-ab86-4bcd-8587-5f8582652a6c/volume-data-source-validator/0.log"
Jul 16 22:40:54.013986 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:54.013952    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rpn85_90027345-b94e-49c5-8d72-0bff02dcdf87/dns/0.log"
Jul 16 22:40:54.047124 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:54.047074    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-rpn85_90027345-b94e-49c5-8d72-0bff02dcdf87/kube-rbac-proxy/0.log"
Jul 16 22:40:54.209266 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:54.209234    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-nj98c_6e9282e2-e6f4-456b-84f7-68872f2c21d0/dns-node-resolver/0.log"
Jul 16 22:40:54.815928 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:54.815888    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-jf8qh_24330e43-7607-4cc4-adb6-57d6475d44a9/node-ca/0.log"
Jul 16 22:40:56.243194 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:56.243168    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-w7rxp_665fbdf6-ab18-40f5-9bc0-5ef816e90bc0/serve-healthcheck-canary/0.log"
Jul 16 22:40:56.760353 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:56.760321    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-4bfcb_2a5960e6-9c85-4d5a-947e-c4075b287904/insights-operator/0.log"
Jul 16 22:40:56.760612 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:56.760592    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-555d9d7885-4bfcb_2a5960e6-9c85-4d5a-947e-c4075b287904/insights-operator/1.log"
Jul 16 22:40:56.956808 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:56.956781    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mfqnh_bf883ff1-b068-427a-b5e7-d69cbf631112/kube-rbac-proxy/0.log"
Jul 16 22:40:57.004970 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:57.004939    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mfqnh_bf883ff1-b068-427a-b5e7-d69cbf631112/exporter/0.log"
Jul 16 22:40:57.037508 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:57.037418    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-mfqnh_bf883ff1-b068-427a-b5e7-d69cbf631112/extractor/0.log"
Jul 16 22:40:58.935792 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:58.935764    2585 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jnrjl/perf-node-gather-daemonset-hbx5f"
Jul 16 22:40:59.488869 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:59.488831    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_kserve-controller-manager-84cf6d888d-gttm2_6a742894-0fa6-49eb-bbb3-f5be3cedc88f/manager/0.log"
Jul 16 22:40:59.835162 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:40:59.835080    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/kserve_seaweedfs-64568bcd49-fskld_bed0b130-8354-4c42-82be-a0190f19c752/seaweedfs/0.log"
Jul 16 22:41:06.229803 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:06.229766    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-86mjg_b5d72dab-5005-4192-809a-00bf8632b0e6/kube-multus/0.log"
Jul 16 22:41:06.694563 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:06.694530    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwr8m_f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b/kube-multus-additional-cni-plugins/0.log"
Jul 16 22:41:06.720111 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:06.720060    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwr8m_f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b/egress-router-binary-copy/0.log"
Jul 16 22:41:06.744807 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:06.744779    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwr8m_f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b/cni-plugins/0.log"
Jul 16 22:41:06.770982 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:06.770951    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwr8m_f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b/bond-cni-plugin/0.log"
Jul 16 22:41:06.801219 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:06.801176    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwr8m_f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b/routeoverride-cni/0.log"
Jul 16 22:41:06.835082 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:06.835052    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwr8m_f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b/whereabouts-cni-bincopy/0.log"
Jul 16 22:41:06.874940 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:06.874909    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-xwr8m_f7eaefc7-e67b-4ee3-857d-dc0f6deabf2b/whereabouts-cni/0.log"
Jul 16 22:41:07.051028 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:07.050999    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l9sj7_3b5ea743-a8b0-4e62-9217-8215b918bee1/network-metrics-daemon/0.log"
Jul 16 22:41:07.081416 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:07.081381    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-l9sj7_3b5ea743-a8b0-4e62-9217-8215b918bee1/kube-rbac-proxy/0.log"
Jul 16 22:41:08.584292 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:08.584257    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnjgb_aa83eae5-eb8c-440d-98ba-5e8f8b3714b2/ovn-controller/0.log"
Jul 16 22:41:08.613115 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:08.613066    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnjgb_aa83eae5-eb8c-440d-98ba-5e8f8b3714b2/ovn-acl-logging/0.log"
Jul 16 22:41:08.657451 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:08.657422    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnjgb_aa83eae5-eb8c-440d-98ba-5e8f8b3714b2/kube-rbac-proxy-node/0.log"
Jul 16 22:41:08.691339 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:08.691299    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnjgb_aa83eae5-eb8c-440d-98ba-5e8f8b3714b2/kube-rbac-proxy-ovn-metrics/0.log"
Jul 16 22:41:08.722647 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:08.722613    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnjgb_aa83eae5-eb8c-440d-98ba-5e8f8b3714b2/northd/0.log"
Jul 16 22:41:08.760145 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:08.760089    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnjgb_aa83eae5-eb8c-440d-98ba-5e8f8b3714b2/nbdb/0.log"
Jul 16 22:41:08.787469 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:08.787439    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnjgb_aa83eae5-eb8c-440d-98ba-5e8f8b3714b2/sbdb/0.log"
Jul 16 22:41:08.845671 ip-10-0-129-202 kubenswrapper[2585]: I0716 22:41:08.845584    2585 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-hnjgb_aa83eae5-eb8c-440d-98ba-5e8f8b3714b2/ovnkube-controller/0.log"